[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:20:55 dist-test-slave-9vg1 krb5kdc[10148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:20:55 dist-test-slave-9vg1 krb5kdc[10148](info): set up 2 sockets
Jun 20 02:20:55 dist-test-slave-9vg1 krb5kdc[10148](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 02:20:56.445060 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:20:56 dist-test-slave-9vg1 krb5kdc[10148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:20:56.636092 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34109
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:20:57.002681 10161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:20:57.003005 10161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:20:57.003207 10161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:20:57.010902 10161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:20:57.011049 10161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:20:57.011189 10161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:20:57.011278 10161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:20:57.011363 10161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:20:57.022691 10161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:20:57.024266 10161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:20:57.025753 10161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:20:57.039427 10166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:57.039327 10169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:57.039363 10167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:20:57.040766 10161 server_base.cc:1034] running on GCE node
I20260620 02:20:57.041622 10161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:20:57.043243 10161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:20:57.044497 10161 hybrid_clock.cc:648] HybridClock initialized: now 1781922057044439 us; error 66 us; skew 500 ppm
I20260620 02:20:57.044916 10161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:20:57 dist-test-slave-9vg1 krb5kdc[10148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922057, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:20:57.048377 10161 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:20:57.050088 10161 webserver.cc:460] Webserver started at http://127.9.231.126:40755/ using document root <none> and password file <none>
I20260620 02:20:57.050841 10161 fs_manager.cc:359] Metadata directory not provided
I20260620 02:20:57.050990 10161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:20:57.051357 10161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:20:57.054595 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "95c1f3f0c60444de9205874ab44b975d"
format_stamp: "Formatted at 2026-06-20 02:20:57 on dist-test-slave-9vg1"
server_key: "c93a741af7b6259df597c47b3501b136"
server_key_iv: "10043c0ae2e3ad0db41f99bd5953a737"
server_key_version: "encryptionkey@0"
I20260620 02:20:57.055472 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "95c1f3f0c60444de9205874ab44b975d"
format_stamp: "Formatted at 2026-06-20 02:20:57 on dist-test-slave-9vg1"
server_key: "c93a741af7b6259df597c47b3501b136"
server_key_iv: "10043c0ae2e3ad0db41f99bd5953a737"
server_key_version: "encryptionkey@0"
I20260620 02:20:57.061715 10161 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 02:20:57.066045 10176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:57.067615 10161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:20:57.067840 10161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "95c1f3f0c60444de9205874ab44b975d"
format_stamp: "Formatted at 2026-06-20 02:20:57 on dist-test-slave-9vg1"
server_key: "c93a741af7b6259df597c47b3501b136"
server_key_iv: "10043c0ae2e3ad0db41f99bd5953a737"
server_key_version: "encryptionkey@0"
I20260620 02:20:57.068063 10161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:20:57.096300 10161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:20:57.101567 10161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:20:57.101971 10161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:20:57.122639 10161 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34109
I20260620 02:20:57.122673 10227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34109 every 8 connection(s)
I20260620 02:20:57.124143 10161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:20:57.126045 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 10161
I20260620 02:20:57.126405 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:20:57.129154 10141 external_mini_cluster.cc:1428] Setting key e3105e30dd9c0fb7dfbdee511f2b9b1c
I20260620 02:20:57.129822 10228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:20:57.142208 10228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d: Bootstrap starting.
I20260620 02:20:57.145996 10228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d: Neither blocks nor log segments found. Creating new log.
I20260620 02:20:57.147419 10228 log.cc:824] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:20:57.150687 10228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d: No bootstrap required, opened a new log
I20260620 02:20:57.158535 10228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c1f3f0c60444de9205874ab44b975d" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34109 } }
I20260620 02:20:57.159016 10228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:20:57.159178 10228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c1f3f0c60444de9205874ab44b975d, State: Initialized, Role: FOLLOWER
I20260620 02:20:57.159883 10228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c1f3f0c60444de9205874ab44b975d" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34109 } }
I20260620 02:20:57.160151 10228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:20:57.160311 10228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:20:57.160516 10228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:20:57.162952 10228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c1f3f0c60444de9205874ab44b975d" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34109 } }
I20260620 02:20:57.163535 10228 leader_election.cc:302] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95c1f3f0c60444de9205874ab44b975d; no voters:
I20260620 02:20:57.164215 10228 leader_election.cc:288] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:20:57.164345 10233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 02:20:57 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:20:57.165817 10233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [term 1 LEADER]: Becoming Leader. State: Replica: 95c1f3f0c60444de9205874ab44b975d, State: Running, Role: LEADER
I20260620 02:20:57.166458 10233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c1f3f0c60444de9205874ab44b975d" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34109 } }
I20260620 02:20:57.166882 10228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:20:57.169797 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95c1f3f0c60444de9205874ab44b975d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c1f3f0c60444de9205874ab44b975d" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34109 } } }
I20260620 02:20:57.170286 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [sys.catalog]: This master's current role is: LEADER
I20260620 02:20:57.171346 10235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c1f3f0c60444de9205874ab44b975d. Latest consensus state: current_term: 1 leader_uuid: "95c1f3f0c60444de9205874ab44b975d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c1f3f0c60444de9205874ab44b975d" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34109 } } }
I20260620 02:20:57.171842 10235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d [sys.catalog]: This master's current role is: LEADER
I20260620 02:20:57.179605 10238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:20:57.184336 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:57.134383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.9.231.126:34109)
0620 02:20:57.135699 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:57.135718 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:57.137341 (+ 1623us) server_negotiation.cc:407] Connection header received
0620 02:20:57.150627 (+ 13286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:57.150653 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:57.151220 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:57.151668 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:57.154943 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:57.156577 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:57.158578 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:57.159791 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:57.160021 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:57.167568 (+ 7547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:57.167579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:57.167597 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:57.167645 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:57.173710 (+ 6065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:57.175468 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:57.175483 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:57.175496 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:57.175808 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:57.176601 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:57.176609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:57.176615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:57.177591 (+ 976us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:57.181172 (+ 3581us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:57.181574 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:20:57.182335 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":720,"threads_started":1}
I20260620 02:20:57.187268 10238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:20:57.200435 10238 catalog_manager.cc:1347] Generated new cluster ID: 0c7f7fc928604eb4a2ef32b6107c8d23
I20260620 02:20:57.200654 10238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:20:57.221084 10238 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:20:57.223193 10238 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:20:57.241155 10238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95c1f3f0c60444de9205874ab44b975d: Generated new TSK 0
I20260620 02:20:57.242365 10238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:20:57.292886 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34109
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:20:57.652310 10254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:20:57.652609 10254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:20:57.652814 10254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:20:57.660758 10254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:20:57.660883 10254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:20:57.661067 10254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:20:57.672467 10254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:20:57.673923 10254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:20:57.675432 10254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:20:57.689612 10259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:57.689692 10260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:57.689862 10262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:20:57.690754 10254 server_base.cc:1034] running on GCE node
I20260620 02:20:57.691476 10254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:20:57.692596 10254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:20:57.693852 10254 hybrid_clock.cc:648] HybridClock initialized: now 1781922057693779 us; error 88 us; skew 500 ppm
I20260620 02:20:57.694176 10254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:20:57 dist-test-slave-9vg1 krb5kdc[10148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922057, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:20:57.697535 10254 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:20:57.699138 10254 webserver.cc:460] Webserver started at http://127.9.231.65:39497/ using document root <none> and password file <none>
I20260620 02:20:57.699867 10254 fs_manager.cc:359] Metadata directory not provided
I20260620 02:20:57.700016 10254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:20:57.700345 10254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:20:57.703547 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "e29e1d85e6b24c388d1c4c471206420f"
format_stamp: "Formatted at 2026-06-20 02:20:57 on dist-test-slave-9vg1"
server_key: "2f4dea266b6fb9f34de772f3a8ea0d4f"
server_key_iv: "0f36817e4fea5da5cc210ed59f64140a"
server_key_version: "encryptionkey@0"
I20260620 02:20:57.704342 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "e29e1d85e6b24c388d1c4c471206420f"
format_stamp: "Formatted at 2026-06-20 02:20:57 on dist-test-slave-9vg1"
server_key: "2f4dea266b6fb9f34de772f3a8ea0d4f"
server_key_iv: "0f36817e4fea5da5cc210ed59f64140a"
server_key_version: "encryptionkey@0"
I20260620 02:20:57.709863 10254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 02:20:57.713861 10269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:57.715610 10254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:20:57.715862 10254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "e29e1d85e6b24c388d1c4c471206420f"
format_stamp: "Formatted at 2026-06-20 02:20:57 on dist-test-slave-9vg1"
server_key: "2f4dea266b6fb9f34de772f3a8ea0d4f"
server_key_iv: "0f36817e4fea5da5cc210ed59f64140a"
server_key_version: "encryptionkey@0"
I20260620 02:20:57.716118 10254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:20:57.734763 10254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:20:57.740350 10254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:20:57.740763 10254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:20:57.741961 10254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:20:57.743878 10254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:20:57.744006 10254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:57.744179 10254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:20:57.744267 10254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:57.784204 10254 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:43925
I20260620 02:20:57.784212 10381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:43925 every 8 connection(s)
I20260620 02:20:57.785638 10254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:20:57.791826 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 10254
I20260620 02:20:57.792136 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:20:57.793000 10141 external_mini_cluster.cc:1428] Setting key 0567c00c414593d967cd58d982c02765
Jun 20 02:20:57 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922057, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:20:57.817616 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:57.788542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:53987 (local address 127.9.231.126:34109)
0620 02:20:57.789040 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:57.789045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:57.790318 (+ 1273us) server_negotiation.cc:407] Connection header received
0620 02:20:57.792566 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:57.792574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:57.792684 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:57.792861 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:57.794618 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:57.795760 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:57.798611 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:57.799519 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:57.799682 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:57.810048 (+ 10366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:57.810054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:57.810058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:57.810096 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:57.813091 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:57.814228 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:57.814233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:57.814236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:57.814364 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:57.814894 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:57.814898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:57.814900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:57.815343 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:57.815525 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:57.816686 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0620 02:20:57.816980 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":277,"threads_started":1}
I20260620 02:20:57.818434 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:57.789226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34109 (local address 127.9.231.65:53987)
0620 02:20:57.790115 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 02:20:57.790171 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:20:57.792256 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:20:57.793133 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:20:57.793152 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:20:57.793687 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:20:57.794434 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:20:57.794452 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:20:57.795964 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:20:57.795970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:20:57.798384 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:20:57.798417 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:20:57.799803 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:20:57.799814 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:20:57.800197 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:57.806562 (+ 6365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:20:57.806594 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:20:57.809701 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:20:57.813307 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:20:57.813315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:20:57.813339 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:20:57.814084 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:20:57.814557 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:20:57.814561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:20:57.814563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:20:57.814778 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:20:57.815639 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:20:57.815650 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:20:57.816175 (+ 525us) client_negotiation.cc:769] Sending connection context
0620 02:20:57.816471 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 02:20:57.817020 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":403,"threads_started":1}
I20260620 02:20:57.820300 10382 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34109
I20260620 02:20:57.820708 10382 heartbeater.cc:464] Registering TS with master...
I20260620 02:20:57.821722 10382 heartbeater.cc:511] Master 127.9.231.126:34109 requested a full tablet report, sending...
I20260620 02:20:57.824293 10193 ts_manager.cc:194] Registered new tserver with Master: e29e1d85e6b24c388d1c4c471206420f (127.9.231.65:43925)
I20260620 02:20:57.825872 10193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:53987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:20:57.839171 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34109
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:20:58.201772 10388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:20:58.202075 10388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:20:58.202181 10388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:20:58.210063 10388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:20:58.210198 10388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:20:58.210318 10388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:20:58.220806 10388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:20:58.222160 10388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:20:58.223567 10388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:20:58.238098 10393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:58.238153 10396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:58.238204 10394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:20:58.239362 10388 server_base.cc:1034] running on GCE node
I20260620 02:20:58.240242 10388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:20:58.241511 10388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:20:58.242759 10388 hybrid_clock.cc:648] HybridClock initialized: now 1781922058242708 us; error 45 us; skew 500 ppm
I20260620 02:20:58.243157 10388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:20:58 dist-test-slave-9vg1 krb5kdc[10148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922058, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:20:58.246627 10388 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:20:58.248194 10388 webserver.cc:460] Webserver started at http://127.9.231.66:46729/ using document root <none> and password file <none>
I20260620 02:20:58.248878 10388 fs_manager.cc:359] Metadata directory not provided
I20260620 02:20:58.248978 10388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:20:58.249260 10388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:20:58.252418 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "31a0ba2350d9445b81ecf3fc6a6918dc"
format_stamp: "Formatted at 2026-06-20 02:20:58 on dist-test-slave-9vg1"
server_key: "e8dc8bbe9759986bfa7a670814f4b1a4"
server_key_iv: "43b3bf7c432e56e02ff13eeb26e51856"
server_key_version: "encryptionkey@0"
I20260620 02:20:58.253185 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "31a0ba2350d9445b81ecf3fc6a6918dc"
format_stamp: "Formatted at 2026-06-20 02:20:58 on dist-test-slave-9vg1"
server_key: "e8dc8bbe9759986bfa7a670814f4b1a4"
server_key_iv: "43b3bf7c432e56e02ff13eeb26e51856"
server_key_version: "encryptionkey@0"
I20260620 02:20:58.258876 10388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:20:58.262588 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:58.264096 10388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:20:58.264310 10388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "31a0ba2350d9445b81ecf3fc6a6918dc"
format_stamp: "Formatted at 2026-06-20 02:20:58 on dist-test-slave-9vg1"
server_key: "e8dc8bbe9759986bfa7a670814f4b1a4"
server_key_iv: "43b3bf7c432e56e02ff13eeb26e51856"
server_key_version: "encryptionkey@0"
I20260620 02:20:58.264489 10388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:20:58.301955 10388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:20:58.307286 10388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:20:58.307610 10388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:20:58.309021 10388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:20:58.310544 10388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:20:58.310631 10388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:58.310803 10388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:20:58.310863 10388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:58.347146 10388 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:45993
I20260620 02:20:58.347258 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:45993 every 8 connection(s)
I20260620 02:20:58.348453 10388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:20:58.358361 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 10388
I20260620 02:20:58.358565 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:20:58.359126 10141 external_mini_cluster.cc:1428] Setting key c2f6a194bd73b241d0504d223ede9b8e
Jun 20 02:20:58 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922058, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:20:58.374137 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:58.351297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46459 (local address 127.9.231.126:34109)
0620 02:20:58.351895 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:58.351899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:58.352967 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 02:20:58.356214 (+ 3247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:58.356218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:58.356291 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:58.356389 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:58.358757 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.359873 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.361994 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.362419 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.362558 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:58.366439 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:58.366448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:58.366455 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:58.366501 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:58.370537 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:58.371518 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:58.371527 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:58.371532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:58.371703 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:58.372163 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:58.372173 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:58.372175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:58.372541 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:58.372724 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:58.373435 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 02:20:58.373676 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":357,"threads_started":1}
I20260620 02:20:58.375253 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:58.351805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34109 (local address 127.9.231.66:46459)
0620 02:20:58.352735 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 02:20:58.352792 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:20:58.355884 (+ 3092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:20:58.356689 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:20:58.356703 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:20:58.357550 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:20:58.358516 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:20:58.358547 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.360056 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.360062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:20:58.361823 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:20:58.361836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.362717 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.362730 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:20:58.362978 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:58.363752 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:20:58.363782 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:20:58.366172 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:20:58.370767 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:20:58.370776 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:20:58.370792 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:20:58.371380 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:20:58.371823 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:20:58.371829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:20:58.371837 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:20:58.372033 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:20:58.372675 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:20:58.372686 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:20:58.373071 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 02:20:58.373352 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 02:20:58.374000 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":419,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:20:58.376838 10516 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34109
I20260620 02:20:58.377228 10516 heartbeater.cc:464] Registering TS with master...
I20260620 02:20:58.378196 10516 heartbeater.cc:511] Master 127.9.231.126:34109 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
I20260620 02:20:58.379943 10193 ts_manager.cc:194] Registered new tserver with Master: 31a0ba2350d9445b81ecf3fc6a6918dc (127.9.231.66:45993)
I20260620 02:20:58.380795 10193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:46459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:20:58.402777 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34109
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:20:58.766995 10522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:20:58.767401 10522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:20:58.767560 10522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:20:58.775674 10522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:20:58.775796 10522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:20:58.775914 10522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:20:58.786363 10522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:20:58.787837 10522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:20:58.789300 10522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:20:58.804629 10530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:58.804780 10527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:20:58.804908 10528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:20:58.806231 10522 server_base.cc:1034] running on GCE node
I20260620 02:20:58.807122 10522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:20:58.808220 10522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:20:58.809592 10522 hybrid_clock.cc:648] HybridClock initialized: now 1781922058809423 us; error 58 us; skew 500 ppm
I20260620 02:20:58.810034 10522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:20:58 dist-test-slave-9vg1 krb5kdc[10148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922058, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:20:58.813715 10522 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:20:58.815444 10522 webserver.cc:460] Webserver started at http://127.9.231.67:37421/ using document root <none> and password file <none>
I20260620 02:20:58.816205 10522 fs_manager.cc:359] Metadata directory not provided
I20260620 02:20:58.816355 10522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:20:58.816689 10522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:20:58.819890 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "23fc4dfb609f439c86059f1832ec7d0a"
format_stamp: "Formatted at 2026-06-20 02:20:58 on dist-test-slave-9vg1"
server_key: "a344f2bfd949b77a845fc74c2cc49b9b"
server_key_iv: "9890f35e6687206295cb189d2bab7c2e"
server_key_version: "encryptionkey@0"
I20260620 02:20:58.820777 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "23fc4dfb609f439c86059f1832ec7d0a"
format_stamp: "Formatted at 2026-06-20 02:20:58 on dist-test-slave-9vg1"
server_key: "a344f2bfd949b77a845fc74c2cc49b9b"
server_key_iv: "9890f35e6687206295cb189d2bab7c2e"
server_key_version: "encryptionkey@0"
I20260620 02:20:58.826939 10522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260620 02:20:58.829823 10382 heartbeater.cc:503] Master 127.9.231.126:34109 was elected leader, sending a full tablet report...
I20260620 02:20:58.830952 10537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:58.832607 10522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:20:58.832758 10522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "23fc4dfb609f439c86059f1832ec7d0a"
format_stamp: "Formatted at 2026-06-20 02:20:58 on dist-test-slave-9vg1"
server_key: "a344f2bfd949b77a845fc74c2cc49b9b"
server_key_iv: "9890f35e6687206295cb189d2bab7c2e"
server_key_version: "encryptionkey@0"
I20260620 02:20:58.832945 10522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:20:58.851229 10522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:20:58.855768 10522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:20:58.856031 10522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:20:58.857004 10522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:20:58.858729 10522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:20:58.858803 10522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:20:58.859195 10522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:20:58.859337 10522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 02:20:58.903302 10522 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:42441
I20260620 02:20:58.903393 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:42441 every 8 connection(s)
I20260620 02:20:58.904655 10522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:20:58.912285 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 10522
I20260620 02:20:58.912493 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:20:58.913100 10141 external_mini_cluster.cc:1428] Setting key 896ed895f3639d50ae75ed6606eeb1b1
Jun 20 02:20:58 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922058, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:20:58.927860 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:58.907625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:56049 (local address 127.9.231.126:34109)
0620 02:20:58.908291 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:58.908296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:58.909220 (+ 924us) server_negotiation.cc:407] Connection header received
0620 02:20:58.911447 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:58.911453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:58.911547 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:58.911670 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:58.913442 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.914402 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.916876 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.917418 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.917576 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:58.921582 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:58.921587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:58.921591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:58.921622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:58.924613 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:58.925334 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:58.925338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:58.925340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:58.925467 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:58.926016 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:58.926021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:58.926024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:58.926291 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:58.926410 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:58.927286 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 02:20:58.927544 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":409,"threads_started":1}
I20260620 02:20:58.928916 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:58.908025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34109 (local address 127.9.231.67:56049)
0620 02:20:58.908839 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 02:20:58.908890 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:20:58.911052 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:20:58.911933 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:20:58.911942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:20:58.912606 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:20:58.913267 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:20:58.913289 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.914640 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.914646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:20:58.916714 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:20:58.916731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.917744 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.917753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:20:58.917987 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:58.918704 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:20:58.918732 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:20:58.921336 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:20:58.924752 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:20:58.924760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:20:58.924775 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:20:58.925222 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:20:58.925687 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:20:58.925691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:20:58.925694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:20:58.925900 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:20:58.926433 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:20:58.926439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:20:58.926815 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 02:20:58.927214 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 02:20:58.927669 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":316,"threads_started":1}
I20260620 02:20:58.930487 10650 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34109
I20260620 02:20:58.930855 10650 heartbeater.cc:464] Registering TS with master...
I20260620 02:20:58.931725 10650 heartbeater.cc:511] Master 127.9.231.126:34109 requested a full tablet report, sending...
I20260620 02:20:58.933532 10193 ts_manager.cc:194] Registered new tserver with Master: 23fc4dfb609f439c86059f1832ec7d0a (127.9.231.67:42441)
I20260620 02:20:58.934367 10193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:56049
I20260620 02:20:58.943351 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:20:58 dist-test-slave-9vg1 krb5kdc[10148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:20:58 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:20:58.993753 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:58.975321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.9.231.65:43925)
0620 02:20:58.976044 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:58.976052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:58.976167 (+ 115us) server_negotiation.cc:407] Connection header received
0620 02:20:58.976328 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:58.976336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:58.976562 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:58.976701 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:58.979010 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.980102 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.981669 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:58.982272 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:58.982373 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:58.987265 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:58.987282 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:58.987299 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:58.987341 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:58.990478 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:58.991305 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:58.991313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:58.991316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:58.991448 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:58.991864 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:58.991868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:58.991870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:58.992262 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:58.992498 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:58.993047 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 02:20:58.993341 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":421,"threads_started":1}
W20260620 02:20:58.995294 10297 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57168
Jun 20 02:20:59 dist-test-slave-9vg1 krb5kdc[10148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:20:59 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:20:59.046160 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:59.030532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.9.231.65:43925)
0620 02:20:59.030820 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:59.030827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:59.031120 (+ 293us) server_negotiation.cc:407] Connection header received
0620 02:20:59.031306 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:59.031311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:59.031378 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:59.031464 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:59.033020 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.033873 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.035410 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.035961 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.036107 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:59.040385 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:59.040390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:59.040394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:59.040498 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:59.043372 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.044082 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.044086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.044088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.044250 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.044644 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.044649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.044651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.044935 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:59.045179 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:59.045440 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 02:20:59.045631 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 02:20:59.048049 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 10254
I20260620 02:20:59.062263 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:59.049241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.9.231.65:43925)
0620 02:20:59.049415 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:59.049419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:59.049917 (+ 498us) server_negotiation.cc:407] Connection header received
0620 02:20:59.050060 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:59.050064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:59.050125 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:59.050206 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:59.051774 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.052491 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.053990 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.054527 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.054624 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:59.057162 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:59.057167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:59.057170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:59.057202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:59.059847 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.060535 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.060538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.060541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.060707 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.061016 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.061019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.061021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.061263 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:59.061435 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:59.061821 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 02:20:59.061999 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:20:59.062952 10377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:20:59.149658 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 10254
I20260620 02:20:59.169826 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 10388
Jun 20 02:20:59 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:20:59.186970 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:59.170837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.9.231.66:45993)
0620 02:20:59.171589 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:59.171597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:59.171623 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:20:59.171696 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:59.171703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:59.171916 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:59.172050 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:59.173468 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.174573 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.175898 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.176532 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.176650 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:59.180864 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:59.180877 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:59.180894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:59.180933 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:59.184170 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.184794 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.184802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.184804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.184943 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.185319 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.185323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.185325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.185767 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:59.186005 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:59.186282 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 02:20:59.186609 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":408,"threads_started":1}
I20260620 02:20:59.188395 10511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:20:59.276062 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 10388
I20260620 02:20:59.300570 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 10522
Jun 20 02:20:59 dist-test-slave-9vg1 krb5kdc[10148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:20:59.317770 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:20:59.301604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.9.231.67:42441)
0620 02:20:59.302260 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 02:20:59.302267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:20:59.302295 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:20:59.302359 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:20:59.302365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:20:59.302598 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:20:59.302734 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:20:59.304045 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.305179 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.306516 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:20:59.307352 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:20:59.307464 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:20:59.311770 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:20:59.311786 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:20:59.311803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:20:59.311846 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:20:59.314830 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.315651 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.315660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.315662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.315785 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:20:59.316167 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:20:59.316171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:20:59.316172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:20:59.316530 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:20:59.316768 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:20:59.317116 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 02:20:59.317327 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":319,"threads_started":1}
I20260620 02:20:59.319288 10645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:20:59.406903 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 10522
I20260620 02:20:59.423772 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 10161
I20260620 02:20:59.425009 10223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:20:59.533336 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 10161
2026-06-20T02:20:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4207 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:20:59 dist-test-slave-9vg1 krb5kdc[10672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:20:59 dist-test-slave-9vg1 krb5kdc[10672](info): set up 2 sockets
Jun 20 02:20:59 dist-test-slave-9vg1 krb5kdc[10672](info): commencing operation
krb5kdc: starting...
W20260620 02:21:00.668926 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:00 dist-test-slave-9vg1 krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:00.871932 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:32989
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:32989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:01.369666 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:01.370299 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:01.370589 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:01.381192 10685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:01.381479 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:01.381613 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:01.381724 10685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:01.381872 10685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:01.395968 10685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:32989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:01.397801 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:01.399448 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:01.416575 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:01.417191 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:01.417460 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:01.419020 10685 server_base.cc:1034] running on GCE node
I20260620 02:21:01.420291 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:01.422096 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:01.423357 10685 hybrid_clock.cc:648] HybridClock initialized: now 1781922061423243 us; error 112 us; skew 500 ppm
I20260620 02:21:01.423828 10685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:01 dist-test-slave-9vg1 krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922061, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:01.428771 10685 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:01.431128 10685 webserver.cc:460] Webserver started at http://127.9.231.126:45055/ using document root <none> and password file <none>
I20260620 02:21:01.431929 10685 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:01.432087 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:01.432433 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:01.435997 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "ff65dc9add544ff7a402bcd25023a777"
format_stamp: "Formatted at 2026-06-20 02:21:01 on dist-test-slave-9vg1"
server_key: "903feddb896298559cf06e38bfd165b5"
server_key_iv: "2251e94ac2ed122da53aa81cb9fce0d2"
server_key_version: "encryptionkey@0"
I20260620 02:21:01.436838 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "ff65dc9add544ff7a402bcd25023a777"
format_stamp: "Formatted at 2026-06-20 02:21:01 on dist-test-slave-9vg1"
server_key: "903feddb896298559cf06e38bfd165b5"
server_key_iv: "2251e94ac2ed122da53aa81cb9fce0d2"
server_key_version: "encryptionkey@0"
I20260620 02:21:01.444104 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260620 02:21:01.448907 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:01.450531 10685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260620 02:21:01.450820 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "ff65dc9add544ff7a402bcd25023a777"
format_stamp: "Formatted at 2026-06-20 02:21:01 on dist-test-slave-9vg1"
server_key: "903feddb896298559cf06e38bfd165b5"
server_key_iv: "2251e94ac2ed122da53aa81cb9fce0d2"
server_key_version: "encryptionkey@0"
I20260620 02:21:01.451058 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:01.482551 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:01.487318 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:01.487654 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:01.519004 10685 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:32989
I20260620 02:21:01.519191 10751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:32989 every 8 connection(s)
I20260620 02:21:01.520643 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:01.525781 10752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:01.527777 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 10685
I20260620 02:21:01.528266 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:01.529987 10141 external_mini_cluster.cc:1428] Setting key ba15c7f1a348b27fb6da441295fb4f9f
I20260620 02:21:01.548435 10752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777: Bootstrap starting.
Jun 20 02:21:01 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:01.553887 10752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:01.555744 10752 log.cc:824] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:01.559753 10752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777: No bootstrap required, opened a new log
I20260620 02:21:01.563246 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:01.533403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.9.231.126:32989)
0620 02:21:01.534559 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:01.534572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:01.534618 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:21:01.536637 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:01.536662 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:01.537111 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:01.537475 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:01.541584 (+ 4109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:01.542954 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:01.545735 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:01.546455 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:01.546592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:01.552583 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:01.552597 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:01.552630 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:01.552687 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:01.557382 (+ 4695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:01.558262 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:01.558272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:01.558280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:01.558509 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:01.558924 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:01.558929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:01.558932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:01.559745 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:01.560074 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:01.560657 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 02:21:01.561157 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":627,"threads_started":1}
I20260620 02:21:01.573377 10752 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65dc9add544ff7a402bcd25023a777" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32989 } }
I20260620 02:21:01.574106 10752 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:01.574338 10752 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff65dc9add544ff7a402bcd25023a777, State: Initialized, Role: FOLLOWER
I20260620 02:21:01.575178 10752 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65dc9add544ff7a402bcd25023a777" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32989 } }
I20260620 02:21:01.575574 10752 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:01.575918 10752 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:01.576205 10752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:01.580106 10752 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65dc9add544ff7a402bcd25023a777" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32989 } }
I20260620 02:21:01.580808 10752 leader_election.cc:302] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff65dc9add544ff7a402bcd25023a777; no voters:
I20260620 02:21:01.582047 10752 leader_election.cc:288] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:01.582326 10757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:01.585026 10757 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [term 1 LEADER]: Becoming Leader. State: Replica: ff65dc9add544ff7a402bcd25023a777, State: Running, Role: LEADER
I20260620 02:21:01.585837 10757 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65dc9add544ff7a402bcd25023a777" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32989 } }
I20260620 02:21:01.586364 10752 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:01.590287 10759 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff65dc9add544ff7a402bcd25023a777. Latest consensus state: current_term: 1 leader_uuid: "ff65dc9add544ff7a402bcd25023a777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65dc9add544ff7a402bcd25023a777" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32989 } } }
I20260620 02:21:01.591320 10759 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:01.591092 10758 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff65dc9add544ff7a402bcd25023a777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65dc9add544ff7a402bcd25023a777" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32989 } } }
I20260620 02:21:01.591598 10758 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:01.597046 10764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:01.617791 10764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:01.633073 10764 catalog_manager.cc:1347] Generated new cluster ID: 7c833fdbd3564c33a3a90425a9f17385
I20260620 02:21:01.633313 10764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:01.650744 10764 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:01.652483 10764 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:01.660652 10764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff65dc9add544ff7a402bcd25023a777: Generated new TSK 0
I20260620 02:21:01.661940 10764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:01.723934 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:32989
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:02.247889 10778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:02.248490 10778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:02.248684 10778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:02.258731 10778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:02.258958 10778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:02.259224 10778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:02.271854 10778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:32989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:02.273406 10778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:02.275009 10778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:02.293180 10783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:02.293269 10784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:02.293252 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:02.294183 10778 server_base.cc:1034] running on GCE node
I20260620 02:21:02.295053 10778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:02.296277 10778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:02.297621 10778 hybrid_clock.cc:648] HybridClock initialized: now 1781922062297472 us; error 42 us; skew 500 ppm
I20260620 02:21:02.297977 10778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:02 dist-test-slave-9vg1 krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922062, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:02.302866 10778 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:02.305035 10778 webserver.cc:460] Webserver started at http://127.9.231.65:44105/ using document root <none> and password file <none>
I20260620 02:21:02.305914 10778 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:02.306095 10778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:02.306460 10778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:02.310756 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "9590c4772c9a4e948730ddb467ae3f04"
format_stamp: "Formatted at 2026-06-20 02:21:02 on dist-test-slave-9vg1"
server_key: "82d9e7c219eaeb1ab56cdd066e74e674"
server_key_iv: "c0173f89e8f7d1ec4a0ff8e5be769eb5"
server_key_version: "encryptionkey@0"
I20260620 02:21:02.311914 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "9590c4772c9a4e948730ddb467ae3f04"
format_stamp: "Formatted at 2026-06-20 02:21:02 on dist-test-slave-9vg1"
server_key: "82d9e7c219eaeb1ab56cdd066e74e674"
server_key_iv: "c0173f89e8f7d1ec4a0ff8e5be769eb5"
server_key_version: "encryptionkey@0"
I20260620 02:21:02.319114 10778 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260620 02:21:02.324833 10793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:02.326740 10778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260620 02:21:02.327020 10778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "9590c4772c9a4e948730ddb467ae3f04"
format_stamp: "Formatted at 2026-06-20 02:21:02 on dist-test-slave-9vg1"
server_key: "82d9e7c219eaeb1ab56cdd066e74e674"
server_key_iv: "c0173f89e8f7d1ec4a0ff8e5be769eb5"
server_key_version: "encryptionkey@0"
I20260620 02:21:02.327373 10778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:02.349843 10778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:02.355934 10778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:02.356310 10778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:02.357893 10778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:02.359756 10778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:02.359864 10778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:02.360118 10778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:02.360249 10778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:02.429862 10778 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:46399
I20260620 02:21:02.430127 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:46399 every 8 connection(s)
I20260620 02:21:02.431692 10778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:02.440958 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 10778
I20260620 02:21:02.441219 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:02.442234 10141 external_mini_cluster.cc:1428] Setting key a8f3cde833c0c1309f46f72c445ecc5e
Jun 20 02:21:02 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922062, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:02.460000 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:02.435645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:45517 (local address 127.9.231.126:32989)
0620 02:21:02.436354 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:02.436360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:02.438062 (+ 1702us) server_negotiation.cc:407] Connection header received
0620 02:21:02.441077 (+ 3015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:02.441086 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:02.441197 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:02.441332 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:02.443674 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:02.444713 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:02.447427 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:02.447865 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:02.448006 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:02.452471 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:02.452476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:02.452479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:02.452515 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:02.455741 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:02.456868 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:02.456873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:02.456875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:02.457023 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:02.457590 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:02.457594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:02.457596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:02.458002 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:02.458176 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:02.459038 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 02:21:02.459355 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":435,"threads_started":1}
I20260620 02:21:02.460963 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:02.436616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:32989 (local address 127.9.231.65:45517)
0620 02:21:02.437831 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:02.437882 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:02.440615 (+ 2733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:02.441687 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:02.441703 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:02.442641 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:02.443488 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:02.443509 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:02.444888 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:02.444894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:02.447264 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:02.447285 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:02.448149 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:02.448159 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:02.448508 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:02.449304 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:02.449346 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:02.452195 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:02.455915 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:02.455929 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:02.455952 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:02.456701 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:02.457198 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:02.457204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:02.457207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:02.457438 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:02.458146 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:02.458155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:02.458605 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 02:21:02.458898 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:21:02.459571 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":498,"threads_started":1}
I20260620 02:21:02.462884 10906 heartbeater.cc:347] Connected to a master server at 127.9.231.126:32989
I20260620 02:21:02.463284 10906 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:21:02.464490 10906 heartbeater.cc:511] Master 127.9.231.126:32989 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:21:02.467309 10717 ts_manager.cc:194] Registered new tserver with Master: 9590c4772c9a4e948730ddb467ae3f04 (127.9.231.65:46399)
I20260620 02:21:02.468847 10717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:45517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:02.490749 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:32989
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:03.024044 10912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:03.024500 10912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:03.024694 10912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:03.038686 10912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:03.038902 10912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:03.039158 10912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:03.052958 10912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:32989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:03.054481 10912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:03.056221 10912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:03.075049 10917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:03.075630 10918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:03.075913 10920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:03.076931 10912 server_base.cc:1034] running on GCE node
I20260620 02:21:03.077785 10912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:03.079169 10912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:03.080513 10912 hybrid_clock.cc:648] HybridClock initialized: now 1781922063080384 us; error 66 us; skew 500 ppm
I20260620 02:21:03.080993 10912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:03 dist-test-slave-9vg1 krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922063, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:03.086198 10912 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:03.088567 10912 webserver.cc:460] Webserver started at http://127.9.231.66:37073/ using document root <none> and password file <none>
I20260620 02:21:03.089552 10912 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:03.089762 10912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:03.090178 10912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:03.094251 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "45d59f70c0d8427f8b000477c952fb53"
format_stamp: "Formatted at 2026-06-20 02:21:03 on dist-test-slave-9vg1"
server_key: "c34fce29eb55db074ea7c534cc683e82"
server_key_iv: "493d950a2951b63cc99e55a194a83518"
server_key_version: "encryptionkey@0"
I20260620 02:21:03.095399 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "45d59f70c0d8427f8b000477c952fb53"
format_stamp: "Formatted at 2026-06-20 02:21:03 on dist-test-slave-9vg1"
server_key: "c34fce29eb55db074ea7c534cc683e82"
server_key_iv: "493d950a2951b63cc99e55a194a83518"
server_key_version: "encryptionkey@0"
I20260620 02:21:03.102357 10912 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260620 02:21:03.107230 10927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:03.108970 10912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:21:03.109179 10912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "45d59f70c0d8427f8b000477c952fb53"
format_stamp: "Formatted at 2026-06-20 02:21:03 on dist-test-slave-9vg1"
server_key: "c34fce29eb55db074ea7c534cc683e82"
server_key_iv: "493d950a2951b63cc99e55a194a83518"
server_key_version: "encryptionkey@0"
I20260620 02:21:03.109462 10912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:03.148243 10912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:03.154066 10912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:03.154497 10912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:03.156054 10912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:03.157936 10912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:03.158056 10912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:03.158232 10912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:03.158321 10912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:03.234714 10912 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:38275
I20260620 02:21:03.234750 11039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:38275 every 8 connection(s)
I20260620 02:21:03.236444 10912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:03.245779 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 10912
I20260620 02:21:03.246003 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:03.246546 10141 external_mini_cluster.cc:1428] Setting key e965e403c17ff12d648def1ee64214a8
Jun 20 02:21:03 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922063, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:03.260726 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.240200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:36763 (local address 127.9.231.126:32989)
0620 02:21:03.241083 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:03.241090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:03.242006 (+ 916us) server_negotiation.cc:407] Connection header received
0620 02:21:03.244188 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:03.244193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:03.244356 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:03.244492 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:03.246680 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.247765 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.249451 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.249981 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.250133 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:03.254405 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:03.254411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:03.254414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:03.254450 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:03.257232 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:03.258137 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:03.258145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:03.258153 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:03.258289 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:03.258763 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:03.258769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:03.258772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:03.259173 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:03.259326 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:03.260089 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 02:21:03.260344 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":571,"threads_started":1}
I20260620 02:21:03.261806 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.240943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:32989 (local address 127.9.231.66:36763)
0620 02:21:03.241808 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:03.241861 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:03.243851 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:03.244739 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:03.244749 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:03.245529 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:03.246444 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:03.246471 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.247943 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.247947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:03.249292 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:03.249305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.250229 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.250236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:03.250476 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:03.251454 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:03.251477 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:03.254128 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:03.257427 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:03.257440 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:03.257462 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:03.257988 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:03.258410 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:03.258415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:03.258419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:03.258644 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:03.259321 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:03.259329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:03.259711 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 02:21:03.260005 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 02:21:03.260590 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":366,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:21:03.263597 11040 heartbeater.cc:347] Connected to a master server at 127.9.231.126:32989
I20260620 02:21:03.263998 11040 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:03.265053 11040 heartbeater.cc:511] Master 127.9.231.126:32989 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
I20260620 02:21:03.266824 10717 ts_manager.cc:194] Registered new tserver with Master: 45d59f70c0d8427f8b000477c952fb53 (127.9.231.66:38275)
I20260620 02:21:03.268177 10717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:36763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:03.291462 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:32989
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:21:03.472741 10906 heartbeater.cc:503] Master 127.9.231.126:32989 was elected leader, sending a full tablet report...
W20260620 02:21:03.744032 11046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:03.744354 11046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:03.744463 11046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:03.752843 11046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:03.752983 11046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:03.753120 11046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:03.766414 11046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:32989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:03.767814 11046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:03.769250 11046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:03.785815 11051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:03.785805 11052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:03.785820 11054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:03.786473 11046 server_base.cc:1034] running on GCE node
I20260620 02:21:03.787501 11046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:03.788619 11046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:03.789880 11046 hybrid_clock.cc:648] HybridClock initialized: now 1781922063789814 us; error 60 us; skew 500 ppm
I20260620 02:21:03.790253 11046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:03 dist-test-slave-9vg1 krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922063, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:03.793911 11046 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:03.795771 11046 webserver.cc:460] Webserver started at http://127.9.231.67:40199/ using document root <none> and password file <none>
I20260620 02:21:03.796617 11046 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:03.796779 11046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:03.797129 11046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:03.800700 11046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "97fe871d92754fbb89cd949ca04cdb94"
format_stamp: "Formatted at 2026-06-20 02:21:03 on dist-test-slave-9vg1"
server_key: "1c6c5ff0844e07462ad0fe69ff5c855d"
server_key_iv: "accfcb559db323107a172c814a0d4b05"
server_key_version: "encryptionkey@0"
I20260620 02:21:03.801681 11046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "97fe871d92754fbb89cd949ca04cdb94"
format_stamp: "Formatted at 2026-06-20 02:21:03 on dist-test-slave-9vg1"
server_key: "1c6c5ff0844e07462ad0fe69ff5c855d"
server_key_iv: "accfcb559db323107a172c814a0d4b05"
server_key_version: "encryptionkey@0"
I20260620 02:21:03.807865 11046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260620 02:21:03.812146 11061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:03.813823 11046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:21:03.814064 11046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "97fe871d92754fbb89cd949ca04cdb94"
format_stamp: "Formatted at 2026-06-20 02:21:03 on dist-test-slave-9vg1"
server_key: "1c6c5ff0844e07462ad0fe69ff5c855d"
server_key_iv: "accfcb559db323107a172c814a0d4b05"
server_key_version: "encryptionkey@0"
I20260620 02:21:03.814323 11046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:03.834913 11046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:03.840004 11046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:03.840399 11046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:03.841554 11046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:03.843257 11046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:03.843361 11046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:03.843559 11046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:03.843611 11046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:03.883984 11046 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:37349
I20260620 02:21:03.884045 11173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:37349 every 8 connection(s)
I20260620 02:21:03.885401 11046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:03.893942 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 11046
I20260620 02:21:03.894179 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:03.894630 10141 external_mini_cluster.cc:1428] Setting key 364675daae642d6c00fad443d576af77
Jun 20 02:21:03 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922063, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:03.909165 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.888289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:55307 (local address 127.9.231.126:32989)
0620 02:21:03.888845 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:03.888849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:03.889941 (+ 1092us) server_negotiation.cc:407] Connection header received
0620 02:21:03.891962 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:03.891968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:03.892073 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:03.892196 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:03.893862 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.894787 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.896769 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.897394 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.897535 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:03.902409 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:03.902414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:03.902417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:03.902456 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:03.905220 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:03.906085 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:03.906090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:03.906093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:03.906241 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:03.906683 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:03.906687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:03.906689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:03.907188 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:03.907347 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:03.908131 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 02:21:03.908438 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":327,"threads_started":1}
I20260620 02:21:03.910130 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.888985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:32989 (local address 127.9.231.67:55307)
0620 02:21:03.889765 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:03.889812 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:03.891641 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:03.892467 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:03.892476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:03.893036 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:03.893651 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:03.893692 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.894957 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.894961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:03.896616 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:03.896631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.897626 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.897633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:03.897921 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:03.898710 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:03.898744 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:03.902130 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:03.905393 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:03.905402 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:03.905418 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:03.905930 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:03.906349 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:03.906352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:03.906354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:03.906566 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:03.907306 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:03.907313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:03.907729 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 02:21:03.907988 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 02:21:03.908525 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":332,"threads_started":1}
I20260620 02:21:03.911716 11174 heartbeater.cc:347] Connected to a master server at 127.9.231.126:32989
I20260620 02:21:03.911999 11174 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:03.912768 11174 heartbeater.cc:511] Master 127.9.231.126:32989 requested a full tablet report, sending...
I20260620 02:21:03.914369 10716 ts_manager.cc:194] Registered new tserver with Master: 97fe871d92754fbb89cd949ca04cdb94 (127.9.231.67:37349)
I20260620 02:21:03.916144 10716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:55307
I20260620 02:21:03.924212 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:21:03.950407 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.934656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.9.231.126:32989)
0620 02:21:03.934890 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:03.934895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:03.935743 (+ 848us) server_negotiation.cc:407] Connection header received
0620 02:21:03.935986 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:03.935991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:03.936085 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:03.936206 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:03.938380 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.939343 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.940888 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.941311 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.941475 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:03.944365 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:03.944370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:03.944373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:03.944416 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:03.947607 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:03.948401 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:03.948405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:03.948408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:03.948565 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:03.948903 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:03.948906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:03.948908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:03.949256 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:03.949452 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:03.949833 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 02:21:03.950024 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:21:03.958547 10716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:03.966434 10716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:04.004990 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.981629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:46399 (local address 127.0.0.1:48440)
0620 02:21:03.994087 (+ 12458us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:03.994161 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:03.994480 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:03.996281 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:03.996288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:03.996321 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:03.996824 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:03.996838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.998908 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.998923 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.001766 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:04.001781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.004180 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.004186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.004374 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.004412 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:21:04.004526 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 02:21:04.004677 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12021,"mutex_wait_us":40,"thread_start_us":13918,"threads_started":1}
I20260620 02:21:04.005399 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.981840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.9.231.65:46399)
0620 02:21:03.995202 (+ 13362us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:03.995212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:03.995246 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:21:03.995353 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:03.995361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:03.995647 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:03.995866 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:03.997061 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.998730 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.001958 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.003618 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.003755 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.004000 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.004614 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.004801 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13148,"thread_start_us":12979,"threads_started":1}
I20260620 02:21:04.006913 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.997043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:37349 (local address 127.0.0.1:53382)
0620 02:21:03.997882 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:03.997902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:03.998022 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:03.998986 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:03.998996 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:03.999020 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:03.999558 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:03.999571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.001237 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.001242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.004061 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:04.004081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.006014 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.006021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.006180 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.006212 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:21:04.006317 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 02:21:04.006459 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":8,"thread_start_us":434,"threads_started":1}
I20260620 02:21:04.007707 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.982049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:38275 (local address 127.0.0.1:55960)
0620 02:21:03.995635 (+ 13586us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:03.995663 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:03.995810 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:03.997146 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:03.997152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:03.997177 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:03.997809 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:03.997823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.999785 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:03.999792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.002176 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:04.002195 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.006309 (+ 4114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.006320 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.006597 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.006629 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:21:04.006731 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:21:04.006933 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13441,"mutex_wait_us":86,"thread_start_us":13042,"threads_started":1}
I20260620 02:21:04.008186 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.982623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.9.231.66:38275)
0620 02:21:03.996272 (+ 13649us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:03.996283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:03.996315 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:21:03.996403 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:03.996408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:03.996794 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:03.997021 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:03.998012 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:03.999615 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.002335 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.003720 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.003859 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.004035 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.006813 (+ 2778us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.007034 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12349,"thread_start_us":13346,"threads_started":1}
I20260620 02:21:04.010555 10841 tablet_service.cc:1467] Processing CreateTablet for tablet ed969a00d3444acab93e7a93a9bd78bd (DEFAULT_TABLE table=test-table [id=b278df006033432ebcd282be5033a60d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:04.012166 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:03.996300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.9.231.67:37349)
0620 02:21:03.997043 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:03.997051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:03.998016 (+ 965us) server_negotiation.cc:407] Connection header received
0620 02:21:03.998162 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:03.998168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:03.998509 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:03.998697 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:03.999790 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.001098 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.004200 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.005800 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.005909 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.006151 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.006381 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.011501 (+ 5120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":401,"threads_started":1}
I20260620 02:21:04.012809 10841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed969a00d3444acab93e7a93a9bd78bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:04.012966 10975 tablet_service.cc:1467] Processing CreateTablet for tablet ed969a00d3444acab93e7a93a9bd78bd (DEFAULT_TABLE table=test-table [id=b278df006033432ebcd282be5033a60d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:04.014707 10975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed969a00d3444acab93e7a93a9bd78bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:04.016991 11109 tablet_service.cc:1467] Processing CreateTablet for tablet ed969a00d3444acab93e7a93a9bd78bd (DEFAULT_TABLE table=test-table [id=b278df006033432ebcd282be5033a60d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:04.019579 11109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed969a00d3444acab93e7a93a9bd78bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:04.033566 11193 tablet_bootstrap.cc:492] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04: Bootstrap starting.
I20260620 02:21:04.037540 11194 tablet_bootstrap.cc:492] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53: Bootstrap starting.
I20260620 02:21:04.038687 11193 tablet_bootstrap.cc:654] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:04.040346 11193 log.cc:824] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:04.041663 11194 tablet_bootstrap.cc:654] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:04.043305 11194 log.cc:824] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:04.043416 11195 tablet_bootstrap.cc:492] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94: Bootstrap starting.
I20260620 02:21:04.047880 11195 tablet_bootstrap.cc:654] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:04.049582 11195 log.cc:824] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:04.054986 11193 tablet_bootstrap.cc:492] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04: No bootstrap required, opened a new log
I20260620 02:21:04.055836 11193 ts_tablet_manager.cc:1397] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.009s
I20260620 02:21:04.058957 11194 tablet_bootstrap.cc:492] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53: No bootstrap required, opened a new log
I20260620 02:21:04.059520 11194 ts_tablet_manager.cc:1397] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.019s
I20260620 02:21:04.060734 11195 tablet_bootstrap.cc:492] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94: No bootstrap required, opened a new log
I20260620 02:21:04.061131 11195 ts_tablet_manager.cc:1397] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260620 02:21:04.069682 11193 raft_consensus.cc:348] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.070263 11193 raft_consensus.cc:374] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:04.070429 11193 raft_consensus.cc:729] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9590c4772c9a4e948730ddb467ae3f04, State: Initialized, Role: FOLLOWER
I20260620 02:21:04.071280 11193 consensus_queue.cc:260] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.071376 11194 raft_consensus.cc:348] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.071893 11194 raft_consensus.cc:374] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:04.072099 11194 raft_consensus.cc:729] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d59f70c0d8427f8b000477c952fb53, State: Initialized, Role: FOLLOWER
I20260620 02:21:04.072966 11194 consensus_queue.cc:260] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.073468 11193 ts_tablet_manager.cc:1428] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04: Time spent starting tablet: real 0.017s user 0.005s sys 0.009s
I20260620 02:21:04.073936 11195 raft_consensus.cc:348] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.074620 11040 heartbeater.cc:503] Master 127.9.231.126:32989 was elected leader, sending a full tablet report...
I20260620 02:21:04.074607 11195 raft_consensus.cc:374] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:04.074736 11195 raft_consensus.cc:729] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fe871d92754fbb89cd949ca04cdb94, State: Initialized, Role: FOLLOWER
I20260620 02:21:04.075459 11194 ts_tablet_manager.cc:1428] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260620 02:21:04.076556 11195 consensus_queue.cc:260] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.078023 11174 heartbeater.cc:503] Master 127.9.231.126:32989 was elected leader, sending a full tablet report...
I20260620 02:21:04.078824 11195 ts_tablet_manager.cc:1428] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260620 02:21:04.137293 11175 tablet.cc:2369] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:04.175468 11200 raft_consensus.cc:482] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:04.175825 11200 raft_consensus.cc:504] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.177634 11200 leader_election.cc:288] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9590c4772c9a4e948730ddb467ae3f04 (127.9.231.65:46399), 97fe871d92754fbb89cd949ca04cdb94 (127.9.231.67:37349)
W20260620 02:21:04.186041 10907 tablet.cc:2369] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:04.186918 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.178173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:52507 (local address 127.9.231.65:46399)
0620 02:21:04.178420 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.178427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.178836 (+ 409us) server_negotiation.cc:407] Connection header received
0620 02:21:04.179466 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.179472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.179570 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.179654 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:04.180345 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.181419 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.184259 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.185649 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.185822 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.185903 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.186448 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.186622 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:21:04.186969 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.177988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:46399 (local address 127.9.231.66:52507)
0620 02:21:04.178700 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:04.178720 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:04.178872 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:04.179711 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:04.179716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:04.179751 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:04.180163 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:04.180172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.181600 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.181603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.184025 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:04.184045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.185897 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.185904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.186133 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.186265 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 02:21:04.186384 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 02:21:04.186554 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":119,"thread_start_us":347,"threads_started":1}
I20260620 02:21:04.187083 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.179306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46325 (local address 127.9.231.67:37349)
0620 02:21:04.179509 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.179514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.179866 (+ 352us) server_negotiation.cc:407] Connection header received
0620 02:21:04.180066 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.180070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.180136 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.180205 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:04.180795 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.181769 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.184287 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.185614 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.185822 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.185903 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.186651 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.186786 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:21:04.187849 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.178800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:37349 (local address 127.9.231.66:46325)
0620 02:21:04.179755 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:04.179774 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:04.179920 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:04.180254 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:04.180261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:04.180291 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:04.180640 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:04.180648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.181908 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.181912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.184140 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:04.184154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.186130 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.186135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:04.186420 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.186468 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 02:21:04.187274 (+ 806us) client_negotiation.cc:240] Negotiation successful
0620 02:21:04.187468 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":43,"thread_start_us":543,"threads_started":1}
I20260620 02:21:04.188354 10861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed969a00d3444acab93e7a93a9bd78bd" candidate_uuid: "45d59f70c0d8427f8b000477c952fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9590c4772c9a4e948730ddb467ae3f04" is_pre_election: true
I20260620 02:21:04.188920 10861 raft_consensus.cc:2454] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d59f70c0d8427f8b000477c952fb53 in term 0.
I20260620 02:21:04.188917 11129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed969a00d3444acab93e7a93a9bd78bd" candidate_uuid: "45d59f70c0d8427f8b000477c952fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fe871d92754fbb89cd949ca04cdb94" is_pre_election: true
I20260620 02:21:04.189425 11129 raft_consensus.cc:2454] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d59f70c0d8427f8b000477c952fb53 in term 0.
I20260620 02:21:04.189685 10930 leader_election.cc:302] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d59f70c0d8427f8b000477c952fb53, 9590c4772c9a4e948730ddb467ae3f04; no voters:
I20260620 02:21:04.190145 11200 raft_consensus.cc:2781] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:04.190305 11200 raft_consensus.cc:482] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:04.190418 11200 raft_consensus.cc:3037] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:04.192766 11200 raft_consensus.cc:504] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.193396 11200 leader_election.cc:288] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [CANDIDATE]: Term 1 election: Requested vote from peers 9590c4772c9a4e948730ddb467ae3f04 (127.9.231.65:46399), 97fe871d92754fbb89cd949ca04cdb94 (127.9.231.67:37349)
I20260620 02:21:04.194067 11129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed969a00d3444acab93e7a93a9bd78bd" candidate_uuid: "45d59f70c0d8427f8b000477c952fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fe871d92754fbb89cd949ca04cdb94"
I20260620 02:21:04.194131 10861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed969a00d3444acab93e7a93a9bd78bd" candidate_uuid: "45d59f70c0d8427f8b000477c952fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9590c4772c9a4e948730ddb467ae3f04"
I20260620 02:21:04.194326 11129 raft_consensus.cc:3037] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:04.194399 10861 raft_consensus.cc:3037] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:04.196960 10861 raft_consensus.cc:2454] T ed969a00d3444acab93e7a93a9bd78bd P 9590c4772c9a4e948730ddb467ae3f04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d59f70c0d8427f8b000477c952fb53 in term 1.
I20260620 02:21:04.196957 11129 raft_consensus.cc:2454] T ed969a00d3444acab93e7a93a9bd78bd P 97fe871d92754fbb89cd949ca04cdb94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d59f70c0d8427f8b000477c952fb53 in term 1.
I20260620 02:21:04.197468 10930 leader_election.cc:302] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d59f70c0d8427f8b000477c952fb53, 9590c4772c9a4e948730ddb467ae3f04; no voters:
I20260620 02:21:04.197793 11200 raft_consensus.cc:2781] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:04.198285 11200 raft_consensus.cc:686] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [term 1 LEADER]: Becoming Leader. State: Replica: 45d59f70c0d8427f8b000477c952fb53, State: Running, Role: LEADER
I20260620 02:21:04.198813 11200 consensus_queue.cc:237] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } }
I20260620 02:21:04.203778 10716 catalog_manager.cc:5495] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45d59f70c0d8427f8b000477c952fb53 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "45d59f70c0d8427f8b000477c952fb53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9590c4772c9a4e948730ddb467ae3f04" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97fe871d92754fbb89cd949ca04cdb94" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d59f70c0d8427f8b000477c952fb53" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38275 } health_report { overall_health: HEALTHY } } }
W20260620 02:21:04.239478 11041 tablet.cc:2369] T ed969a00d3444acab93e7a93a9bd78bd P 45d59f70c0d8427f8b000477c952fb53: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:04.262247 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.244168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.9.231.65:46399)
0620 02:21:04.244500 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.244508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.244860 (+ 352us) server_negotiation.cc:407] Connection header received
0620 02:21:04.245142 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.245147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.245228 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.245349 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:04.247451 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.248323 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.249817 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.250291 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.250462 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.255493 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:04.255516 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:04.255534 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:04.255583 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:04.258906 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.259797 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.259806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.259808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.259986 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.260405 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.260409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.260411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.260837 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:04.261066 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.261761 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.261946 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:04.315979 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.299181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.9.231.65:46399)
0620 02:21:04.299435 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.299440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.299567 (+ 127us) server_negotiation.cc:407] Connection header received
0620 02:21:04.299823 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.299828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.299942 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.300042 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:04.301814 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.302816 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.304234 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.304693 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.304803 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.309460 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:04.309469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:04.309474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:04.309522 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:04.312880 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.313701 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.313706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.313709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.313866 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.314349 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.314353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.314356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.314648 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:04.314852 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.315345 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.315601 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:04.364130 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.349361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.9.231.65:46399)
0620 02:21:04.349710 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.349716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.349900 (+ 184us) server_negotiation.cc:407] Connection header received
0620 02:21:04.350065 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.350069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.350134 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.350221 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:04.351816 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.352653 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.353914 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.354400 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.354524 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.358672 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:04.358678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:04.358681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:04.358723 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:04.361433 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.362272 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.362276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.362278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.362397 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.362761 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.362763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.362765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.363049 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:04.363383 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.363673 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.363855 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:21:04.365994 10821 tablet_service.cc:2908] Rejecting scan request for tablet ed969a00d3444acab93e7a93a9bd78bd: Uninitialized: clean time has not yet been initialized
I20260620 02:21:04.370383 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 10778
I20260620 02:21:04.385183 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.371504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.9.231.65:46399)
0620 02:21:04.371705 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.371713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.372172 (+ 459us) server_negotiation.cc:407] Connection header received
0620 02:21:04.372307 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.372310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.372389 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.372476 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:04.373884 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.374694 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.376403 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.376852 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.376968 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.379720 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:04.379727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:04.379733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:04.379780 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:04.382358 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.383213 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.383218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.383221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.383346 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.383763 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.383766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.383768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.384055 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:04.384271 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.384675 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.384851 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:21:04.385926 10901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:04.479794 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 10778
I20260620 02:21:04.499553 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 10912
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:21:04.515753 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.500540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.9.231.66:38275)
0620 02:21:04.500727 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.500732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.500750 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:21:04.500847 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.500851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.500915 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.500999 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:04.502316 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.503156 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.504679 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.505091 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.505206 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.509625 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:04.509643 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:04.509660 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:04.509701 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:04.512749 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.513498 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.513506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.513508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.513695 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.514135 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.514139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.514141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.514576 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:04.514858 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.515211 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.515475 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:21:04.516521 11035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:04.609586 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 10912
I20260620 02:21:04.627533 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11046
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:21:04.644014 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:04.628417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.9.231.67:37349)
0620 02:21:04.628716 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:04.628723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:04.628743 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:21:04.628980 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:04.628984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:04.629051 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:04.629170 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:04.630447 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.631284 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.632687 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:04.633093 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:04.633235 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:04.637845 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:04.637860 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:04.637878 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:04.637917 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:04.641075 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.641833 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.641842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.641844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.642009 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:04.642434 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:04.642438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:04.642440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:04.642800 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:04.643048 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:04.643446 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 02:21:04.643622 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260620 02:21:04.644834 11169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:04.735220 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 11046
I20260620 02:21:04.752029 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 10685
I20260620 02:21:04.753302 10747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:04.867959 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 10685
2026-06-20T02:21:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5341 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[11223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[11223](info): set up 2 sockets
Jun 20 02:21:04 dist-test-slave-9vg1 krb5kdc[11223](info): commencing operation
krb5kdc: starting...
W20260620 02:21:05.980480 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:06 dist-test-slave-9vg1 krb5kdc[11223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:06.187486 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44385
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:06.531391 11236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:06.531780 11236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:06.531967 11236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:06.539587 11236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:06.539773 11236 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:21:06.539849 11236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:06.539950 11236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:06.540026 11236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:06.540117 11236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:06.550449 11236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:44385
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:06.551995 11236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:06.553418 11236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:06.565914 11241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:06.565973 11244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:06.565984 11242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:06.566679 11236 server_base.cc:1034] running on GCE node
I20260620 02:21:06.567613 11236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:06.569154 11236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:06.570446 11236 hybrid_clock.cc:648] HybridClock initialized: now 1781922066570366 us; error 44 us; skew 500 ppm
I20260620 02:21:06.570860 11236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:06 dist-test-slave-9vg1 krb5kdc[11223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922066, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:06.574491 11236 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:06.576503 11236 webserver.cc:460] Webserver started at http://127.9.231.126:33095/ using document root <none> and password file <none>
I20260620 02:21:06.577289 11236 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:06.577430 11236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:06.577787 11236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:06.581069 11236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "4992cbd9220b44dc956ce55da9c6752c"
format_stamp: "Formatted at 2026-06-20 02:21:06 on dist-test-slave-9vg1"
server_key: "cec1e5863ac158ee8fbcd523a6eb217a"
server_key_iv: "efd222f356647786594292c566f8308c"
server_key_version: "encryptionkey@0"
I20260620 02:21:06.581930 11236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "4992cbd9220b44dc956ce55da9c6752c"
format_stamp: "Formatted at 2026-06-20 02:21:06 on dist-test-slave-9vg1"
server_key: "cec1e5863ac158ee8fbcd523a6eb217a"
server_key_iv: "efd222f356647786594292c566f8308c"
server_key_version: "encryptionkey@0"
I20260620 02:21:06.588115 11236 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 02:21:06.592335 11251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:06.594130 11236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:21:06.594297 11236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "4992cbd9220b44dc956ce55da9c6752c"
format_stamp: "Formatted at 2026-06-20 02:21:06 on dist-test-slave-9vg1"
server_key: "cec1e5863ac158ee8fbcd523a6eb217a"
server_key_iv: "efd222f356647786594292c566f8308c"
server_key_version: "encryptionkey@0"
I20260620 02:21:06.594506 11236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:06.625754 11236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:06.631134 11236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:06.631485 11236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:06.656522 11236 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:44385
I20260620 02:21:06.656546 11312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:44385 every 8 connection(s)
I20260620 02:21:06.658182 11236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:06.662317 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:06.668198 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 11236
I20260620 02:21:06.668546 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:06.669317 10141 external_mini_cluster.cc:1428] Setting key e4ebcfac10eb72c4a596ff098cc10b50
I20260620 02:21:06.674858 11313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c: Bootstrap starting.
I20260620 02:21:06.679143 11313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:06.680563 11313 log.cc:824] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:06.684559 11313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c: No bootstrap required, opened a new log
Jun 20 02:21:06 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:06.693645 11313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4992cbd9220b44dc956ce55da9c6752c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44385 } }
I20260620 02:21:06.694157 11313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:06.694290 11313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4992cbd9220b44dc956ce55da9c6752c, State: Initialized, Role: FOLLOWER
I20260620 02:21:06.695039 11313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4992cbd9220b44dc956ce55da9c6752c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44385 } }
I20260620 02:21:06.695434 11313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:06.695602 11313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:06.695798 11313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:06.697332 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:06.671852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.9.231.126:44385)
0620 02:21:06.672809 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:06.672826 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:06.672880 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:21:06.675432 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:06.675466 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:06.676058 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:06.676613 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:06.678107 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:06.680181 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:06.682084 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:06.683228 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:06.683425 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:06.688401 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:06.688411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:06.688430 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:06.688480 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:06.692319 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:06.693264 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:06.693274 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:06.693282 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:06.693519 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:06.694006 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:06.694010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:06.694013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:06.694767 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:06.694990 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:06.695781 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 02:21:06.696259 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":374,"threads_started":1}
I20260620 02:21:06.698557 11313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4992cbd9220b44dc956ce55da9c6752c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44385 } }
I20260620 02:21:06.699209 11313 leader_election.cc:302] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4992cbd9220b44dc956ce55da9c6752c; no voters:
I20260620 02:21:06.699875 11313 leader_election.cc:288] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:06.700016 11318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:06.701316 11318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [term 1 LEADER]: Becoming Leader. State: Replica: 4992cbd9220b44dc956ce55da9c6752c, State: Running, Role: LEADER
I20260620 02:21:06.701802 11318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4992cbd9220b44dc956ce55da9c6752c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44385 } }
I20260620 02:21:06.702072 11313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:06.704766 11320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4992cbd9220b44dc956ce55da9c6752c. Latest consensus state: current_term: 1 leader_uuid: "4992cbd9220b44dc956ce55da9c6752c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4992cbd9220b44dc956ce55da9c6752c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44385 } } }
I20260620 02:21:06.704651 11319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4992cbd9220b44dc956ce55da9c6752c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4992cbd9220b44dc956ce55da9c6752c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44385 } } }
I20260620 02:21:06.705297 11319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:06.705323 11320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:06.714841 11324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:06.720701 11324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:06.729770 11324 catalog_manager.cc:1347] Generated new cluster ID: 754bb54a26704f44b2df2251cc0d7dfa
I20260620 02:21:06.729913 11324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:06.748605 11324 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:06.750299 11324 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:06.760980 11324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c: Generated new TSK 0
I20260620 02:21:06.761863 11324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:06.820554 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44385
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:07.173023 11339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:07.173326 11339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:07.173549 11339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:07.181057 11339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:07.181186 11339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:07.181352 11339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:21:07.181470 11339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:07.191664 11339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:44385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:07.193073 11339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:07.194447 11339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:07.208446 11345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:07.208478 11344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:07.208606 11347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:07.209987 11339 server_base.cc:1034] running on GCE node
I20260620 02:21:07.210685 11339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:07.211724 11339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:07.212956 11339 hybrid_clock.cc:648] HybridClock initialized: now 1781922067212883 us; error 46 us; skew 500 ppm
I20260620 02:21:07.213325 11339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:07 dist-test-slave-9vg1 krb5kdc[11223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922067, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:07.217206 11339 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:07.219129 11339 webserver.cc:460] Webserver started at http://127.9.231.65:35143/ using document root <none> and password file <none>
I20260620 02:21:07.219907 11339 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:07.220047 11339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:07.220377 11339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:07.223605 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "e883ebba2be3497da2caa1c13261f0a5"
format_stamp: "Formatted at 2026-06-20 02:21:07 on dist-test-slave-9vg1"
server_key: "603aabdb0d9dd4e41c8d6de175977e14"
server_key_iv: "534e3e6ea7d0096c321b423a23c92c09"
server_key_version: "encryptionkey@0"
I20260620 02:21:07.224416 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "e883ebba2be3497da2caa1c13261f0a5"
format_stamp: "Formatted at 2026-06-20 02:21:07 on dist-test-slave-9vg1"
server_key: "603aabdb0d9dd4e41c8d6de175977e14"
server_key_iv: "534e3e6ea7d0096c321b423a23c92c09"
server_key_version: "encryptionkey@0"
I20260620 02:21:07.230629 11339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260620 02:21:07.234874 11354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:07.236579 11339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:07.236811 11339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "e883ebba2be3497da2caa1c13261f0a5"
format_stamp: "Formatted at 2026-06-20 02:21:07 on dist-test-slave-9vg1"
server_key: "603aabdb0d9dd4e41c8d6de175977e14"
server_key_iv: "534e3e6ea7d0096c321b423a23c92c09"
server_key_version: "encryptionkey@0"
I20260620 02:21:07.237133 11339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:07.258644 11339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:07.263762 11339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:07.264076 11339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:07.267143 11339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:07.267314 11339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 02:21:07.267501 11339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:07.267585 11339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:21:07 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922067, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:07.293226 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.267510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:60163 (local address 127.9.231.126:44385)
0620 02:21:07.270131 (+ 2621us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:07.270140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:07.270163 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:21:07.271751 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:07.271757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:07.271852 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:07.271983 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:07.274320 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.275848 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.278673 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.279191 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.279338 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.284377 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:07.284386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:07.284391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:07.284497 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:07.288212 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.289786 (+ 1574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.289794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.289800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.289960 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.290624 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.290634 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.290641 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.291190 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:07.291364 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:07.292593 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0620 02:21:07.292838 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2468,"thread_start_us":2332,"threads_started":1}
I20260620 02:21:07.294499 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.268242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44385 (local address 127.9.231.65:60163)
0620 02:21:07.269147 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:07.269197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:07.271446 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:07.272242 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:07.272251 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:07.273068 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:07.274000 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.274028 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.276076 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.276083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.278486 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.278501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.279512 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.279520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.279845 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.280652 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:07.280676 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:07.284007 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:07.288465 (+ 4458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.288482 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.288523 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.289591 (+ 1068us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.290184 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.290188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.290191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.290472 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.291367 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:07.291378 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:07.292051 (+ 673us) client_negotiation.cc:769] Sending connection context
0620 02:21:07.292497 (+ 446us) client_negotiation.cc:240] Negotiation successful
0620 02:21:07.293104 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":376,"threads_started":1}
I20260620 02:21:07.324242 11339 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:35189
I20260620 02:21:07.324257 11474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:35189 every 8 connection(s)
I20260620 02:21:07.325556 11339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:07.327978 11475 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44385
I20260620 02:21:07.328243 11475 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:07.329110 11475 heartbeater.cc:511] Master 127.9.231.126:44385 requested a full tablet report, sending...
I20260620 02:21:07.330497 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.314782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:33835 (local address 127.9.231.126:44385)
0620 02:21:07.314990 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:07.314995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:07.315950 (+ 955us) server_negotiation.cc:407] Connection header received
0620 02:21:07.316219 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:07.316225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:07.316312 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:07.316439 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:07.317621 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.318776 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.321371 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.321957 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.322098 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.324299 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:07.324306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:07.324312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:07.324381 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:07.327500 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.328328 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.328336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.328346 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.328492 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.328895 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.328899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.328901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.329224 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:07.329403 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:07.329873 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 02:21:07.330109 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:21:07.330551 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.314871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44385 (local address 127.9.231.65:33835)
0620 02:21:07.315817 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:07.315836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:07.316044 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:07.316475 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:07.316480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:07.317057 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:07.317446 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.317460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.318972 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.318977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.321117 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.321132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.322314 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.322318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.322437 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.323216 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:07.323249 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:07.324087 (+ 838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:07.327639 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.327645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.327649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.328173 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.328610 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.328614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.328618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.328775 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.329362 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:07.329367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:07.329561 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 02:21:07.329761 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 02:21:07.330118 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":595,"threads_started":1}
I20260620 02:21:07.331399 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 11339
I20260620 02:21:07.331642 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:07.331573 11268 ts_manager.cc:194] Registered new tserver with Master: e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65:35189)
I20260620 02:21:07.332579 10141 external_mini_cluster.cc:1428] Setting key 4a1081f127b7fece36a747cb5fbd543e
I20260620 02:21:07.333168 11268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:60163
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:07.375916 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44385
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:07.713557 11479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:07.713855 11479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:07.714020 11479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:07.721724 11479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:07.721853 11479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:07.722028 11479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:21:07.722174 11479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:07.732543 11479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:44385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:07.733966 11479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:07.735478 11479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:07.750277 11487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:07.750355 11485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:07.750244 11484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:07.752665 11479 server_base.cc:1034] running on GCE node
I20260620 02:21:07.753356 11479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:07.754418 11479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:07.755677 11479 hybrid_clock.cc:648] HybridClock initialized: now 1781922067755596 us; error 51 us; skew 500 ppm
I20260620 02:21:07.756085 11479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:07 dist-test-slave-9vg1 krb5kdc[11223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922067, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:07.759806 11479 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:07.761659 11479 webserver.cc:460] Webserver started at http://127.9.231.66:36175/ using document root <none> and password file <none>
I20260620 02:21:07.762435 11479 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:07.762569 11479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:07.762900 11479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:07.765987 11479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "123d2b632b054c2184afacdc5c8cb209"
format_stamp: "Formatted at 2026-06-20 02:21:07 on dist-test-slave-9vg1"
server_key: "c5cc42b3b3d8daf0068de0d53e772178"
server_key_iv: "59481be69f66a39811a244899b223412"
server_key_version: "encryptionkey@0"
I20260620 02:21:07.766819 11479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "123d2b632b054c2184afacdc5c8cb209"
format_stamp: "Formatted at 2026-06-20 02:21:07 on dist-test-slave-9vg1"
server_key: "c5cc42b3b3d8daf0068de0d53e772178"
server_key_iv: "59481be69f66a39811a244899b223412"
server_key_version: "encryptionkey@0"
I20260620 02:21:07.772503 11479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:21:07.776413 11494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:07.778007 11479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:07.778229 11479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "123d2b632b054c2184afacdc5c8cb209"
format_stamp: "Formatted at 2026-06-20 02:21:07 on dist-test-slave-9vg1"
server_key: "c5cc42b3b3d8daf0068de0d53e772178"
server_key_iv: "59481be69f66a39811a244899b223412"
server_key_version: "encryptionkey@0"
I20260620 02:21:07.778558 11479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:07.807848 11479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:07.813165 11479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:07.813510 11479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:07.816926 11479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:07.817056 11479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:07.817245 11479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:07.817332 11479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:21:07 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922067, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:07.852264 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.817363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:43685 (local address 127.9.231.126:44385)
0620 02:21:07.817561 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:07.817566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:07.819728 (+ 2162us) server_negotiation.cc:407] Connection header received
0620 02:21:07.822119 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:07.822125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:07.822227 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:07.822347 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:07.824916 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.826433 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.829094 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.829997 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.830176 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.842647 (+ 12471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:07.842660 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:07.842669 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:07.842739 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:07.847809 (+ 5070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.849253 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.849264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.849271 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.849459 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.849974 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.849980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.849983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.850460 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:07.850664 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:07.851651 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 02:21:07.851902 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:21:07.853330 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.818347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44385 (local address 127.9.231.66:43685)
0620 02:21:07.819512 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:07.819565 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:07.821763 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:07.822700 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:07.822719 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:07.823678 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:07.824610 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.824644 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.826670 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.826681 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.828902 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.828917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.830307 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.830317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.830663 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.838173 (+ 7510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:07.838202 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:07.842248 (+ 4046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:07.848154 (+ 5906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.848171 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.848200 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.849065 (+ 865us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.849604 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.849608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.849610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.849825 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.850646 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:07.850657 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:07.851137 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 02:21:07.851459 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 02:21:07.852088 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":537,"threads_started":1}
I20260620 02:21:07.869087 11479 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:41809
I20260620 02:21:07.869136 11613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:41809 every 8 connection(s)
I20260620 02:21:07.870987 11479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:07.874054 11614 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44385
I20260620 02:21:07.874302 11614 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:07.875003 11614 heartbeater.cc:511] Master 127.9.231.126:44385 requested a full tablet report, sending...
I20260620 02:21:07.876453 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 11479
I20260620 02:21:07.876732 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:07.876881 11267 ts_manager.cc:194] Registered new tserver with Master: 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66:41809)
I20260620 02:21:07.877313 10141 external_mini_cluster.cc:1428] Setting key efe6689999f2f0da2ca7caff145d0b52
I20260620 02:21:07.878127 11267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:43685
I20260620 02:21:07.880265 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.862420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:52197 (local address 127.9.231.126:44385)
0620 02:21:07.862620 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:07.862624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:07.863564 (+ 940us) server_negotiation.cc:407] Connection header received
0620 02:21:07.863846 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:07.863851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:07.863954 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:07.864071 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:07.866165 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.867650 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.869636 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.870306 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.870472 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.873356 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:07.873366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:07.873371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:07.873419 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:07.877035 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.877890 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.877897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.877901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.878092 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:07.878510 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:07.878516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:07.878520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:07.878991 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:07.879190 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:07.879681 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 02:21:07.879919 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:21:07.880360 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:07.862560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44385 (local address 127.9.231.66:52197)
0620 02:21:07.863432 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:07.863461 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:07.863694 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:07.864144 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:07.864153 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:07.865142 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:07.865869 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.865892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.867847 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.867853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.869471 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:07.869487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:07.870536 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:07.870541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:07.870685 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:07.871480 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:07.871521 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:07.873082 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:07.877183 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.877189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.877193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.877743 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.878218 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:07.878223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:07.878226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:07.878389 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:07.879168 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:07.879173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:07.879381 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 02:21:07.879553 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 02:21:07.879883 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":1,"thread_start_us":407,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:07.922634 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44385
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:08.265622 11618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:08.265940 11618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:08.266109 11618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:08.273399 11618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:08.273530 11618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:08.273689 11618 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:21:08.273809 11618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:08.284575 11618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:44385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:08.286029 11618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:08.287669 11618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:08.303429 11623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:08.303467 11624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:08.303544 11626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:08.304340 11618 server_base.cc:1034] running on GCE node
I20260620 02:21:08.305080 11618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:08.306097 11618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:08.307328 11618 hybrid_clock.cc:648] HybridClock initialized: now 1781922068307226 us; error 97 us; skew 500 ppm
I20260620 02:21:08.307744 11618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:08 dist-test-slave-9vg1 krb5kdc[11223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922068, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:08.311295 11618 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:08.313071 11618 webserver.cc:460] Webserver started at http://127.9.231.67:46141/ using document root <none> and password file <none>
I20260620 02:21:08.313831 11618 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:08.313972 11618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:08.314316 11618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:08.317507 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "bf1fcdfdccc34ce4a02ddb09588481bb"
format_stamp: "Formatted at 2026-06-20 02:21:08 on dist-test-slave-9vg1"
server_key: "24d788cb5acb46e7f65abf1dfae459d3"
server_key_iv: "bc5307dc4dd48a578ce2b0005f8a1c8f"
server_key_version: "encryptionkey@0"
I20260620 02:21:08.318348 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "bf1fcdfdccc34ce4a02ddb09588481bb"
format_stamp: "Formatted at 2026-06-20 02:21:08 on dist-test-slave-9vg1"
server_key: "24d788cb5acb46e7f65abf1dfae459d3"
server_key_iv: "bc5307dc4dd48a578ce2b0005f8a1c8f"
server_key_version: "encryptionkey@0"
I20260620 02:21:08.323958 11618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:21:08.327868 11633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:08.329423 11618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:08.329610 11618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "bf1fcdfdccc34ce4a02ddb09588481bb"
format_stamp: "Formatted at 2026-06-20 02:21:08 on dist-test-slave-9vg1"
server_key: "24d788cb5acb46e7f65abf1dfae459d3"
server_key_iv: "bc5307dc4dd48a578ce2b0005f8a1c8f"
server_key_version: "encryptionkey@0"
I20260620 02:21:08.329834 11618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:08.336727 11475 heartbeater.cc:503] Master 127.9.231.126:44385 was elected leader, sending a full tablet report...
I20260620 02:21:08.353834 11618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:08.358733 11618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:08.359030 11618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:08.361959 11618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:08.362075 11618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:08.362232 11618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:08.362316 11618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:21:08 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922068, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:08.387920 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.362720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:60873 (local address 127.9.231.126:44385)
0620 02:21:08.362935 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.362941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.364466 (+ 1525us) server_negotiation.cc:407] Connection header received
0620 02:21:08.366572 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.366577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.366670 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.366795 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:08.369218 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.370703 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.373413 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.374231 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.374389 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.379978 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:08.379984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:08.379988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:08.380032 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:08.383997 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:08.385017 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:08.385023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:08.385028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:08.385204 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:08.385748 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:08.385753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:08.385757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:08.386088 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:08.386269 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.387197 (+ 928us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.387504 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:21:08.389617 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.363453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44385 (local address 127.9.231.67:60873)
0620 02:21:08.364290 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.364328 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.366272 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.367150 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.367165 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.368062 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:08.368944 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.368997 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.370897 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.370908 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.373239 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.373258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.374512 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.374520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.374741 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.375702 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:08.375742 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:08.379593 (+ 3851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:08.384188 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:08.384200 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:08.384235 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:08.384843 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:08.385340 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:08.385345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:08.385350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:08.385599 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:08.386241 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:08.386250 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:08.386701 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.387007 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.387694 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":338,"threads_started":1}
I20260620 02:21:08.415805 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.398008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44385 (local address 127.9.231.67:50457)
0620 02:21:08.398837 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.398875 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.399106 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.399770 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.399775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.400364 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:08.400682 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.400697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.401827 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.401835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.404001 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.404026 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.405261 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.405289 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.405483 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.406228 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:08.406248 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:08.407427 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:08.412170 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:08.412175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:08.412179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:08.412840 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:08.413438 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:08.413442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:08.413445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:08.413602 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:08.414380 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:08.414386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:08.414683 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.414860 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.415436 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":451,"threads_started":1}
I20260620 02:21:08.416348 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.397927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:50457 (local address 127.9.231.126:44385)
0620 02:21:08.398185 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.398194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.398999 (+ 805us) server_negotiation.cc:407] Connection header received
0620 02:21:08.399356 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.399365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.399606 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.399732 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:08.400857 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.401646 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.404258 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.404885 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.404983 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.407738 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:08.407745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:08.407778 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:08.407831 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:08.411951 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:08.413045 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:08.413053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:08.413059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:08.413256 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:08.413823 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:08.413830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:08.413836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:08.414197 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:08.414412 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.415496 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.415824 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 02:21:08.427277 11618 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:34983
I20260620 02:21:08.427312 11752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:34983 every 8 connection(s)
I20260620 02:21:08.428620 11618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:08.430984 11753 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44385
I20260620 02:21:08.431262 11753 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:08.431945 11753 heartbeater.cc:511] Master 127.9.231.126:44385 requested a full tablet report, sending...
I20260620 02:21:08.432819 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 11618
I20260620 02:21:08.433040 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:08.433616 10141 external_mini_cluster.cc:1428] Setting key 0efda2e170e16ccddc709537d0ce73f9
I20260620 02:21:08.433737 11268 ts_manager.cc:194] Registered new tserver with Master: bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67:34983)
I20260620 02:21:08.434872 11268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:60873
I20260620 02:21:08.436785 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:21:08 dist-test-slave-9vg1 krb5kdc[11223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:21:08 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:08.486809 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.471209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.9.231.126:44385)
0620 02:21:08.471443 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.471451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.471731 (+ 280us) server_negotiation.cc:407] Connection header received
0620 02:21:08.471976 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.471989 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.472056 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.472165 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:08.473982 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.474741 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.476208 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.476664 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.476819 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.481052 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:08.481086 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:08.481089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:08.481129 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:08.484204 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:08.484954 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:08.484958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:08.484961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:08.485111 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:08.485542 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:08.485545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:08.485547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:08.485830 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:08.486017 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.486306 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.486488 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:21:08.491657 11268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:08.498975 11268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:08.525344 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.512441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.9.231.66:41809)
0620 02:21:08.513463 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.513477 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.513513 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:21:08.513654 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.513667 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.514004 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.514280 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:08.515174 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.517142 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.521014 (+ 3872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.523104 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.523268 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.523581 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.524564 (+ 983us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.524827 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":823,"threads_started":1}
I20260620 02:21:08.526859 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.512033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:41809 (local address 127.0.0.1:45272)
0620 02:21:08.512865 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.512916 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.513098 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.514301 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.514308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.514340 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:08.514895 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.514913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.517390 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.517398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.520745 (+ 3347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.520768 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.523823 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.523832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.524185 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.524266 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.526014 (+ 1748us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.526346 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":567,"threads_started":1}
I20260620 02:21:08.530778 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.512413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34983 (local address 127.0.0.1:57626)
0620 02:21:08.515036 (+ 2623us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.516385 (+ 1349us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.516545 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.517766 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.517774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.517809 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:08.518441 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.518459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.523048 (+ 4589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.523083 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.527872 (+ 4789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.527897 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.529885 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.529891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.530071 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.530107 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.530179 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.530323 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2465,"thread_start_us":2275,"threads_started":1}
I20260620 02:21:08.531905 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.513121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.9.231.67:34983)
0620 02:21:08.514011 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.514017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.516710 (+ 2693us) server_negotiation.cc:407] Connection header received
0620 02:21:08.516846 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.516858 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.517240 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.517473 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:08.520521 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.522804 (+ 2283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.528080 (+ 5276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.529616 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.531035 (+ 1419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.531323 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.531413 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.531576 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":568,"threads_started":1}
I20260620 02:21:08.531611 11544 tablet_service.cc:1467] Processing CreateTablet for tablet 8c178f5188eb451fb5c1297ff235793f (DEFAULT_TABLE table=test-table [id=aa45e8c8a4ea4d8f82f94ebb62deb479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:08.534188 11544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c178f5188eb451fb5c1297ff235793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:08.535888 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.515266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.9.231.65:35189)
0620 02:21:08.528517 (+ 13251us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.528531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.528556 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:21:08.528645 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.528651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.528838 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.529144 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:08.530037 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.531189 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.533058 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.534604 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.534763 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.534977 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.535404 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.535583 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13095,"thread_start_us":12968,"threads_started":1}
I20260620 02:21:08.535830 11683 tablet_service.cc:1467] Processing CreateTablet for tablet 8c178f5188eb451fb5c1297ff235793f (DEFAULT_TABLE table=test-table [id=aa45e8c8a4ea4d8f82f94ebb62deb479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:08.536796 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.514842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35189 (local address 127.0.0.1:39824)
0620 02:21:08.527608 (+ 12766us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.527628 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.527760 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.529386 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.529392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.529418 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:08.529820 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.529834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.531322 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.531327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.532858 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.532876 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.534886 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.534893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.535130 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.535167 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.535266 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.536422 (+ 1156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12622,"mutex_wait_us":195,"thread_start_us":12001,"threads_started":1}
I20260620 02:21:08.537392 11683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c178f5188eb451fb5c1297ff235793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:08.540791 11405 tablet_service.cc:1467] Processing CreateTablet for tablet 8c178f5188eb451fb5c1297ff235793f (DEFAULT_TABLE table=test-table [id=aa45e8c8a4ea4d8f82f94ebb62deb479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:08.542466 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c178f5188eb451fb5c1297ff235793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:08.547331 11772 tablet_bootstrap.cc:492] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Bootstrap starting.
I20260620 02:21:08.550520 11772 tablet_bootstrap.cc:654] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:08.551743 11772 log.cc:824] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:08.554440 11773 tablet_bootstrap.cc:492] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Bootstrap starting.
I20260620 02:21:08.557696 11772 tablet_bootstrap.cc:492] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: No bootstrap required, opened a new log
I20260620 02:21:08.558158 11772 ts_tablet_manager.cc:1397] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260620 02:21:08.558728 11773 tablet_bootstrap.cc:654] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:08.560087 11773 log.cc:824] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:08.561918 11775 tablet_bootstrap.cc:492] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Bootstrap starting.
I20260620 02:21:08.566953 11775 tablet_bootstrap.cc:654] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:08.568477 11775 log.cc:824] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:08.569008 11772 raft_consensus.cc:348] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.569528 11772 raft_consensus.cc:374] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:08.569696 11772 raft_consensus.cc:729] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123d2b632b054c2184afacdc5c8cb209, State: Initialized, Role: FOLLOWER
I20260620 02:21:08.570366 11772 consensus_queue.cc:260] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.570608 11773 tablet_bootstrap.cc:492] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: No bootstrap required, opened a new log
I20260620 02:21:08.571031 11773 ts_tablet_manager.cc:1397] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260620 02:21:08.571460 11614 heartbeater.cc:503] Master 127.9.231.126:44385 was elected leader, sending a full tablet report...
I20260620 02:21:08.571705 11775 tablet_bootstrap.cc:492] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: No bootstrap required, opened a new log
I20260620 02:21:08.571997 11775 ts_tablet_manager.cc:1397] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 02:21:08.573518 11772 ts_tablet_manager.cc:1428] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 02:21:08.583645 11773 raft_consensus.cc:348] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.583683 11775 raft_consensus.cc:348] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.584219 11773 raft_consensus.cc:374] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:08.584249 11775 raft_consensus.cc:374] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:08.584374 11775 raft_consensus.cc:729] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e883ebba2be3497da2caa1c13261f0a5, State: Initialized, Role: FOLLOWER
I20260620 02:21:08.584353 11773 raft_consensus.cc:729] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1fcdfdccc34ce4a02ddb09588481bb, State: Initialized, Role: FOLLOWER
I20260620 02:21:08.585170 11775 consensus_queue.cc:260] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.585168 11773 consensus_queue.cc:260] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.586508 11753 heartbeater.cc:503] Master 127.9.231.126:44385 was elected leader, sending a full tablet report...
I20260620 02:21:08.587419 11773 ts_tablet_manager.cc:1428] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260620 02:21:08.587635 11775 ts_tablet_manager.cc:1428] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260620 02:21:08.623600 11615 tablet.cc:2369] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:08.667922 11780 raft_consensus.cc:482] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:08.668265 11780 raft_consensus.cc:504] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.670054 11780 leader_election.cc:288] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66:41809), bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67:34983)
I20260620 02:21:08.677163 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.670579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:38065 (local address 127.9.231.67:34983)
0620 02:21:08.670921 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.670929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.671662 (+ 733us) server_negotiation.cc:407] Connection header received
0620 02:21:08.671861 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.671871 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.671935 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.672040 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:08.672621 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.673356 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.675286 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.676138 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.676342 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.676431 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.676751 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.676898 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260620 02:21:08.677276 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.670792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:55161 (local address 127.9.231.66:41809)
0620 02:21:08.670990 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.670997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.671326 (+ 329us) server_negotiation.cc:407] Connection header received
0620 02:21:08.671588 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.671592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.671655 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.671764 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:08.672358 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.673147 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.675272 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.676117 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.676340 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.676421 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.676752 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.676896 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:21:08.677704 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.670484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34983 (local address 127.9.231.65:38065)
0620 02:21:08.671553 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.671567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.671719 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.672053 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.672057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.672086 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:08.672470 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.672479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.673505 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.673511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.675047 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.675118 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.676421 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.676427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.676593 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.676616 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.676723 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.676882 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":16,"thread_start_us":746,"threads_started":1}
I20260620 02:21:08.677743 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.670294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:41809 (local address 127.9.231.65:55161)
0620 02:21:08.671207 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.671224 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.671424 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.671783 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.671787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.671814 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:08.672207 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.672217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.673300 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.673305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.675048 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.675115 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.676364 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.676371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.676568 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.676608 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.676722 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.676898 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":176,"thread_start_us":568,"threads_started":1}
I20260620 02:21:08.678791 11708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c178f5188eb451fb5c1297ff235793f" candidate_uuid: "e883ebba2be3497da2caa1c13261f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" is_pre_election: true
I20260620 02:21:08.678807 11564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c178f5188eb451fb5c1297ff235793f" candidate_uuid: "e883ebba2be3497da2caa1c13261f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123d2b632b054c2184afacdc5c8cb209" is_pre_election: true
I20260620 02:21:08.679337 11564 raft_consensus.cc:2454] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e883ebba2be3497da2caa1c13261f0a5 in term 0.
I20260620 02:21:08.679340 11708 raft_consensus.cc:2454] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e883ebba2be3497da2caa1c13261f0a5 in term 0.
I20260620 02:21:08.680083 11356 leader_election.cc:302] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123d2b632b054c2184afacdc5c8cb209, e883ebba2be3497da2caa1c13261f0a5; no voters:
W20260620 02:21:08.680224 11754 tablet.cc:2369] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:08.680560 11780 raft_consensus.cc:2781] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:08.680735 11780 raft_consensus.cc:482] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:08.680866 11780 raft_consensus.cc:3037] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:08.683434 11780 raft_consensus.cc:504] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.684093 11780 leader_election.cc:288] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66:41809), bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67:34983)
I20260620 02:21:08.684618 11708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c178f5188eb451fb5c1297ff235793f" candidate_uuid: "e883ebba2be3497da2caa1c13261f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb"
I20260620 02:21:08.684630 11564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c178f5188eb451fb5c1297ff235793f" candidate_uuid: "e883ebba2be3497da2caa1c13261f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123d2b632b054c2184afacdc5c8cb209"
I20260620 02:21:08.684834 11708 raft_consensus.cc:3037] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:08.684857 11564 raft_consensus.cc:3037] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:08.687227 11564 raft_consensus.cc:2454] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e883ebba2be3497da2caa1c13261f0a5 in term 1.
I20260620 02:21:08.687244 11708 raft_consensus.cc:2454] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e883ebba2be3497da2caa1c13261f0a5 in term 1.
I20260620 02:21:08.687788 11355 leader_election.cc:302] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 123d2b632b054c2184afacdc5c8cb209, bf1fcdfdccc34ce4a02ddb09588481bb, e883ebba2be3497da2caa1c13261f0a5; no voters:
I20260620 02:21:08.688216 11780 raft_consensus.cc:2781] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:08.688807 11780 raft_consensus.cc:686] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: e883ebba2be3497da2caa1c13261f0a5, State: Running, Role: LEADER
I20260620 02:21:08.689354 11780 consensus_queue.cc:237] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } }
I20260620 02:21:08.694379 11268 catalog_manager.cc:5495] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65). New cstate: current_term: 1 leader_uuid: "e883ebba2be3497da2caa1c13261f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } health_report { overall_health: HEALTHY } } }
I20260620 02:21:08.725423 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.719122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.9.231.65:35189)
0620 02:21:08.719461 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.719469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.719570 (+ 101us) server_negotiation.cc:407] Connection header received
0620 02:21:08.719945 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.719951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.720039 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.720188 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:08.720567 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.721260 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.722655 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.723123 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.723265 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.724058 (+ 793us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:08.724793 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:08.724944 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.725135 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.725210 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 02:21:08.734799 11708 raft_consensus.cc:1261] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Refusing update from remote peer e883ebba2be3497da2caa1c13261f0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:08.734869 11564 raft_consensus.cc:1261] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Refusing update from remote peer e883ebba2be3497da2caa1c13261f0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:08.735998 11783 consensus_queue.cc:1035] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:08.736490 11780 consensus_queue.cc:1035] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:08.749565 11786 mvcc.cc:205] Tried to move back new op lower bound from 7298752793527119872 to 7298752793355386880. Current Snapshot: MvccSnapshot[applied={T|T < 7298752793527119872}]
I20260620 02:21:08.752326 11788 mvcc.cc:205] Tried to move back new op lower bound from 7298752793527119872 to 7298752793355386880. Current Snapshot: MvccSnapshot[applied={T|T < 7298752793527119872}]
I20260620 02:21:08.776940 11267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:08.777208 11267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:08.781526 11267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c: Sending DeleteTablet for 3 replicas of tablet 8c178f5188eb451fb5c1297ff235793f
I20260620 02:21:08.783245 11683 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c178f5188eb451fb5c1297ff235793f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:08 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:57626
I20260620 02:21:08.783416 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c178f5188eb451fb5c1297ff235793f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:08 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:39824
I20260620 02:21:08.782996 11544 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c178f5188eb451fb5c1297ff235793f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:08 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260620 02:21:08.788938 11267 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:08.789958 11267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:08.797623 11798 tablet_replica.cc:330] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: stopping tablet replica
I20260620 02:21:08.798393 11798 raft_consensus.cc:2229] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:08.799674 11798 raft_consensus.cc:2258] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:08.800756 11797 tablet_replica.cc:330] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: stopping tablet replica
I20260620 02:21:08.801317 11797 raft_consensus.cc:2229] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:08.801966 11797 raft_consensus.cc:2258] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:08.803350 11544 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76c56de2fe49a5ad95fac9ca498022 (DEFAULT_TABLE table=test-table [id=399381fb18104adbaac0de396db051d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:08.803508 11682 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76c56de2fe49a5ad95fac9ca498022 (DEFAULT_TABLE table=test-table [id=399381fb18104adbaac0de396db051d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:08.803828 11544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76c56de2fe49a5ad95fac9ca498022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:08.804171 11682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76c56de2fe49a5ad95fac9ca498022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:08.804668 11797 ts_tablet_manager.cc:1905] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:08.810045 11798 ts_tablet_manager.cc:1905] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:08.811703 11772 tablet_bootstrap.cc:492] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: Bootstrap starting.
I20260620 02:21:08.812193 11796 tablet_replica.cc:330] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: stopping tablet replica
I20260620 02:21:08.812775 11796 raft_consensus.cc:2229] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:08.813608 11772 tablet_bootstrap.cc:654] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:08.813613 11796 raft_consensus.cc:2258] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:08.816212 11405 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76c56de2fe49a5ad95fac9ca498022 (DEFAULT_TABLE table=test-table [id=399381fb18104adbaac0de396db051d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:08.816867 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76c56de2fe49a5ad95fac9ca498022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:08.818063 11797 ts_tablet_manager.cc:1918] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:08.818305 11797 log.cc:1196] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/8c178f5188eb451fb5c1297ff235793f
I20260620 02:21:08.818746 11773 tablet_bootstrap.cc:492] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: Bootstrap starting.
I20260620 02:21:08.820441 11773 tablet_bootstrap.cc:654] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:08.824011 11796 ts_tablet_manager.cc:1905] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:08.825965 11775 tablet_bootstrap.cc:492] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: Bootstrap starting.
I20260620 02:21:08.827525 11798 ts_tablet_manager.cc:1918] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:08.827823 11798 log.cc:1196] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/8c178f5188eb451fb5c1297ff235793f
I20260620 02:21:08.828421 11775 tablet_bootstrap.cc:654] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:08.828562 11798 ts_tablet_manager.cc:1939] T 8c178f5188eb451fb5c1297ff235793f P 123d2b632b054c2184afacdc5c8cb209: Deleting consensus metadata
I20260620 02:21:08.831189 11797 ts_tablet_manager.cc:1939] T 8c178f5188eb451fb5c1297ff235793f P bf1fcdfdccc34ce4a02ddb09588481bb: Deleting consensus metadata
I20260620 02:21:08.832338 11253 catalog_manager.cc:4841] TS 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66:41809): tablet 8c178f5188eb451fb5c1297ff235793f (table test-table [id=aa45e8c8a4ea4d8f82f94ebb62deb479]) successfully deleted
I20260620 02:21:08.833455 11252 catalog_manager.cc:4841] TS bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67:34983): tablet 8c178f5188eb451fb5c1297ff235793f (table test-table [id=aa45e8c8a4ea4d8f82f94ebb62deb479]) successfully deleted
I20260620 02:21:08.837095 11772 tablet_bootstrap.cc:492] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: No bootstrap required, opened a new log
I20260620 02:21:08.837319 11772 ts_tablet_manager.cc:1397] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20260620 02:21:08.838462 11772 raft_consensus.cc:348] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.838801 11772 raft_consensus.cc:374] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:08.838936 11772 raft_consensus.cc:729] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123d2b632b054c2184afacdc5c8cb209, State: Initialized, Role: FOLLOWER
I20260620 02:21:08.839403 11772 consensus_queue.cc:260] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.841917 11796 ts_tablet_manager.cc:1918] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:08.842166 11796 log.cc:1196] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/8c178f5188eb451fb5c1297ff235793f
I20260620 02:21:08.842478 11772 ts_tablet_manager.cc:1428] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:21:08.844866 11773 tablet_bootstrap.cc:492] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: No bootstrap required, opened a new log
I20260620 02:21:08.846248 11773 ts_tablet_manager.cc:1397] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.006s
I20260620 02:21:08.848284 11773 raft_consensus.cc:348] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.848687 11773 raft_consensus.cc:374] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:08.848937 11773 raft_consensus.cc:729] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1fcdfdccc34ce4a02ddb09588481bb, State: Initialized, Role: FOLLOWER
I20260620 02:21:08.849820 11773 consensus_queue.cc:260] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.852229 11773 ts_tablet_manager.cc:1428] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260620 02:21:08.853688 11796 ts_tablet_manager.cc:1939] T 8c178f5188eb451fb5c1297ff235793f P e883ebba2be3497da2caa1c13261f0a5: Deleting consensus metadata
I20260620 02:21:08.854643 11775 tablet_bootstrap.cc:492] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: No bootstrap required, opened a new log
I20260620 02:21:08.855830 11254 catalog_manager.cc:4841] TS e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65:35189): tablet 8c178f5188eb451fb5c1297ff235793f (table test-table [id=aa45e8c8a4ea4d8f82f94ebb62deb479]) successfully deleted
I20260620 02:21:08.856079 11775 ts_tablet_manager.cc:1397] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.013s
I20260620 02:21:08.857545 11775 raft_consensus.cc:348] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.857738 11775 raft_consensus.cc:374] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:08.857864 11775 raft_consensus.cc:729] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e883ebba2be3497da2caa1c13261f0a5, State: Initialized, Role: FOLLOWER
I20260620 02:21:08.858142 11775 consensus_queue.cc:260] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.858830 11775 ts_tablet_manager.cc:1428] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260620 02:21:08.941257 11779 raft_consensus.cc:482] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:08.941519 11779 raft_consensus.cc:504] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.943387 11779 leader_election.cc:288] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65:35189), 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66:41809)
I20260620 02:21:08.952183 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.944052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:60985 (local address 127.9.231.66:41809)
0620 02:21:08.944233 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.944240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.944967 (+ 727us) server_negotiation.cc:407] Connection header received
0620 02:21:08.945214 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.945220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.945306 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.945387 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:08.946152 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.946884 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.949676 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.951019 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.951228 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.951310 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.951712 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.951852 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:21:08.953030 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.943651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:41809 (local address 127.9.231.67:60985)
0620 02:21:08.944832 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.944859 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.944991 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.945560 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.945563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.945587 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:08.945968 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.945980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.947019 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.947026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.949472 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.949489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.951279 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.951288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.951498 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.951535 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.952482 (+ 947us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.952643 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":121,"thread_start_us":651,"threads_started":1}
I20260620 02:21:08.953816 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.943875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:43223 (local address 127.9.231.65:35189)
0620 02:21:08.944148 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:08.944157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:08.944757 (+ 600us) server_negotiation.cc:407] Connection header received
0620 02:21:08.945098 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:08.945103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:08.945230 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:08.945332 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:08.945899 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.947118 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.951403 (+ 4285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.952698 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.952877 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.952951 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:08.953372 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 02:21:08.953517 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 02:21:08.953830 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:08.943611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35189 (local address 127.9.231.67:43223)
0620 02:21:08.944634 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:08.944658 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:08.944879 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:08.945368 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:08.945374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:08.945402 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:08.945635 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.945668 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.947237 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.947242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.949472 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:08.949489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:08.952936 (+ 3447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:08.952944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:08.953165 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:08.953205 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:21:08.953325 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:21:08.953473 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":29,"thread_start_us":983,"threads_started":1}
I20260620 02:21:08.953765 11563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76c56de2fe49a5ad95fac9ca498022" candidate_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123d2b632b054c2184afacdc5c8cb209" is_pre_election: true
I20260620 02:21:08.954090 11563 raft_consensus.cc:2454] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1fcdfdccc34ce4a02ddb09588481bb in term 0.
I20260620 02:21:08.954634 11635 leader_election.cc:302] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123d2b632b054c2184afacdc5c8cb209, bf1fcdfdccc34ce4a02ddb09588481bb; no voters:
I20260620 02:21:08.954807 11429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76c56de2fe49a5ad95fac9ca498022" candidate_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e883ebba2be3497da2caa1c13261f0a5" is_pre_election: true
I20260620 02:21:08.955046 11779 raft_consensus.cc:2781] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:08.955195 11779 raft_consensus.cc:482] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:08.955256 11779 raft_consensus.cc:3037] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:08.955317 11429 raft_consensus.cc:2454] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1fcdfdccc34ce4a02ddb09588481bb in term 0.
I20260620 02:21:08.957693 11779 raft_consensus.cc:504] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.958364 11779 leader_election.cc:288] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [CANDIDATE]: Term 1 election: Requested vote from peers e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65:35189), 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66:41809)
I20260620 02:21:08.958874 11429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76c56de2fe49a5ad95fac9ca498022" candidate_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e883ebba2be3497da2caa1c13261f0a5"
I20260620 02:21:08.959018 11563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76c56de2fe49a5ad95fac9ca498022" candidate_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123d2b632b054c2184afacdc5c8cb209"
I20260620 02:21:08.959141 11429 raft_consensus.cc:3037] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:08.959270 11563 raft_consensus.cc:3037] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:08.962468 11563 raft_consensus.cc:2454] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1fcdfdccc34ce4a02ddb09588481bb in term 1.
I20260620 02:21:08.962544 11429 raft_consensus.cc:2454] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1fcdfdccc34ce4a02ddb09588481bb in term 1.
I20260620 02:21:08.963003 11636 leader_election.cc:302] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf1fcdfdccc34ce4a02ddb09588481bb, e883ebba2be3497da2caa1c13261f0a5; no voters:
I20260620 02:21:08.963480 11779 raft_consensus.cc:2781] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:08.964006 11779 raft_consensus.cc:686] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 LEADER]: Becoming Leader. State: Replica: bf1fcdfdccc34ce4a02ddb09588481bb, State: Running, Role: LEADER
I20260620 02:21:08.964648 11779 consensus_queue.cc:237] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:08.968657 11268 catalog_manager.cc:5495] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67). New cstate: current_term: 1 leader_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } health_report { overall_health: HEALTHY } } }
Jun 20 02:21:09 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922066, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:09.021895 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.005768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44385 (local address 127.0.0.1:52678)
0620 02:21:09.006646 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:09.006672 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:09.006866 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:09.007276 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:09.007279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:09.008241 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:09.008496 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.008507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.009674 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.009681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.010733 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.010746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.011678 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.011685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.011876 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.012657 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:09.012674 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:09.015508 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:09.019082 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:09.019100 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:09.019110 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:09.019618 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:09.020039 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:09.020045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:09.020049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:09.020177 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:09.020724 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:09.020732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:09.020918 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 02:21:09.021089 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 02:21:09.021448 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":550,"threads_started":1}
I20260620 02:21:09.021993 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.005702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.9.231.126:44385)
0620 02:21:09.006507 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.006513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.006774 (+ 261us) server_negotiation.cc:407] Connection header received
0620 02:21:09.007004 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.007009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.007140 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.007252 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:09.008661 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.009527 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.010934 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.011477 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.011630 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.015793 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:09.015798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:09.015801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:09.015834 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:09.018872 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.019775 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.019785 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.019788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.019925 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.020288 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.020293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.020297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.020584 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:09.020741 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.021174 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.021599 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":600,"thread_start_us":363,"threads_started":1}
I20260620 02:21:09.029017 11268 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:52678:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:21:09.030340 11268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:09.038715 11405 tablet_service.cc:1467] Processing CreateTablet for tablet 38fb0247aeb04fc3aa7d59843e678013 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62142b683fec490780a4a7c8e203cde4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:21:09.039439 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fb0247aeb04fc3aa7d59843e678013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:09.039637 11544 tablet_service.cc:1467] Processing CreateTablet for tablet 38fb0247aeb04fc3aa7d59843e678013 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62142b683fec490780a4a7c8e203cde4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:21:09.040172 11544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fb0247aeb04fc3aa7d59843e678013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:09.043108 11682 tablet_service.cc:1467] Processing CreateTablet for tablet 38fb0247aeb04fc3aa7d59843e678013 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62142b683fec490780a4a7c8e203cde4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:21:09.043735 11682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fb0247aeb04fc3aa7d59843e678013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:09.046923 11772 tablet_bootstrap.cc:492] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: Bootstrap starting.
I20260620 02:21:09.049142 11772 tablet_bootstrap.cc:654] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:09.055624 11773 tablet_bootstrap.cc:492] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: Bootstrap starting.
I20260620 02:21:09.056707 11775 tablet_bootstrap.cc:492] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: Bootstrap starting.
I20260620 02:21:09.057359 11772 tablet_bootstrap.cc:492] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: No bootstrap required, opened a new log
I20260620 02:21:09.057505 11773 tablet_bootstrap.cc:654] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:09.057591 11772 ts_tablet_manager.cc:1397] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260620 02:21:09.058598 11772 raft_consensus.cc:348] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.058831 11775 tablet_bootstrap.cc:654] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:09.058864 11772 raft_consensus.cc:374] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:09.058955 11772 raft_consensus.cc:729] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123d2b632b054c2184afacdc5c8cb209, State: Initialized, Role: FOLLOWER
I20260620 02:21:09.059283 11772 consensus_queue.cc:260] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.059784 11778 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.060173 11778 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:09.064312 11772 ts_tablet_manager.cc:1428] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260620 02:21:09.069929 11775 tablet_bootstrap.cc:492] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: No bootstrap required, opened a new log
I20260620 02:21:09.069927 11773 tablet_bootstrap.cc:492] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: No bootstrap required, opened a new log
I20260620 02:21:09.070144 11775 ts_tablet_manager.cc:1397] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260620 02:21:09.070138 11773 ts_tablet_manager.cc:1397] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260620 02:21:09.070995 11775 raft_consensus.cc:348] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.070979 11773 raft_consensus.cc:348] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.071292 11773 raft_consensus.cc:374] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:09.071300 11775 raft_consensus.cc:374] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:09.071374 11773 raft_consensus.cc:729] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1fcdfdccc34ce4a02ddb09588481bb, State: Initialized, Role: FOLLOWER
I20260620 02:21:09.071378 11775 raft_consensus.cc:729] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e883ebba2be3497da2caa1c13261f0a5, State: Initialized, Role: FOLLOWER
I20260620 02:21:09.071662 11773 consensus_queue.cc:260] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.071668 11775 consensus_queue.cc:260] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.072384 11775 ts_tablet_manager.cc:1428] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:21:09.072170 11793 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.072649 11793 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:09.073514 11779 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.073833 11779 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:09.074366 11773 ts_tablet_manager.cc:1428] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260620 02:21:09.078424 11476 tablet.cc:2369] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:09.149786 11778 raft_consensus.cc:482] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:09.150054 11778 raft_consensus.cc:504] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.152012 11778 leader_election.cc:288] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65:35189), bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67:34983)
I20260620 02:21:09.162009 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.152492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:41145 (local address 127.9.231.67:34983)
0620 02:21:09.152869 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.152880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.153889 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 02:21:09.154152 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.154160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.154273 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.154396 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:09.155266 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.156425 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.158993 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.160496 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.160700 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.160818 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.161403 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.161582 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260620 02:21:09.162408 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.152299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35189 (local address 127.9.231.66:58543)
0620 02:21:09.153481 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:09.153507 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:09.153773 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:09.154187 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:09.154191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:09.154224 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:09.154709 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.154725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.156463 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.156469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.159146 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.159166 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.161406 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.161413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.161597 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.161623 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:21:09.161752 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 02:21:09.161921 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":84,"thread_start_us":1207,"threads_started":1}
I20260620 02:21:09.162509 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.152786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:58543 (local address 127.9.231.65:35189)
0620 02:21:09.153012 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.153019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.153628 (+ 609us) server_negotiation.cc:407] Connection header received
0620 02:21:09.153968 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.153974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.154066 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.154180 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:09.154918 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.156300 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.159408 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.161115 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.161354 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.161446 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.161794 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.161967 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:21:09.163309 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.152438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34983 (local address 127.9.231.66:41145)
0620 02:21:09.153782 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:09.153798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:09.153991 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:09.154439 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:09.154444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:09.154474 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:09.155017 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.155035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.156579 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.156585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.158785 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.158803 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.160911 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.160921 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.161139 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.161180 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:21:09.162512 (+ 1332us) client_negotiation.cc:240] Negotiation successful
0620 02:21:09.162716 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"mutex_wait_us":30,"thread_start_us":635,"threads_started":1}
I20260620 02:21:09.163203 11429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fb0247aeb04fc3aa7d59843e678013" candidate_uuid: "123d2b632b054c2184afacdc5c8cb209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e883ebba2be3497da2caa1c13261f0a5" is_pre_election: true
I20260620 02:21:09.163614 11429 raft_consensus.cc:2454] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123d2b632b054c2184afacdc5c8cb209 in term 0.
I20260620 02:21:09.164237 11497 leader_election.cc:302] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123d2b632b054c2184afacdc5c8cb209, e883ebba2be3497da2caa1c13261f0a5; no voters:
I20260620 02:21:09.164135 11707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fb0247aeb04fc3aa7d59843e678013" candidate_uuid: "123d2b632b054c2184afacdc5c8cb209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" is_pre_election: true
I20260620 02:21:09.164546 11707 raft_consensus.cc:2454] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123d2b632b054c2184afacdc5c8cb209 in term 0.
I20260620 02:21:09.164752 11778 raft_consensus.cc:2781] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:09.164938 11778 raft_consensus.cc:482] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:09.165068 11778 raft_consensus.cc:3037] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:09.168393 11778 raft_consensus.cc:504] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.169150 11778 leader_election.cc:288] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [CANDIDATE]: Term 1 election: Requested vote from peers e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65:35189), bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67:34983)
I20260620 02:21:09.169782 11429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fb0247aeb04fc3aa7d59843e678013" candidate_uuid: "123d2b632b054c2184afacdc5c8cb209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e883ebba2be3497da2caa1c13261f0a5"
I20260620 02:21:09.169986 11707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fb0247aeb04fc3aa7d59843e678013" candidate_uuid: "123d2b632b054c2184afacdc5c8cb209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb"
I20260620 02:21:09.170116 11429 raft_consensus.cc:3037] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:09.170315 11707 raft_consensus.cc:3037] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:09.173779 11429 raft_consensus.cc:2454] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123d2b632b054c2184afacdc5c8cb209 in term 1.
I20260620 02:21:09.174376 11497 leader_election.cc:302] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123d2b632b054c2184afacdc5c8cb209, e883ebba2be3497da2caa1c13261f0a5; no voters:
I20260620 02:21:09.174818 11778 raft_consensus.cc:2781] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:09.175135 11707 raft_consensus.cc:2454] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123d2b632b054c2184afacdc5c8cb209 in term 1.
I20260620 02:21:09.178733 11778 raft_consensus.cc:686] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [term 1 LEADER]: Becoming Leader. State: Replica: 123d2b632b054c2184afacdc5c8cb209, State: Running, Role: LEADER
I20260620 02:21:09.179661 11778 consensus_queue.cc:237] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } }
I20260620 02:21:09.182489 11818 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: TxnStatusTablet state changed. Reason: New leader 123d2b632b054c2184afacdc5c8cb209. Latest consensus state: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.182955 11818 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:21:09.183792 11820 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:21:09.184106 11820 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:21:09.185946 11268 catalog_manager.cc:5495] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } health_report { overall_health: UNKNOWN } } }
I20260620 02:21:09.218530 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.212500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.9.231.66:41809)
0620 02:21:09.212699 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.212706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.212736 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:21:09.212992 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.212997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.213107 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.213293 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:09.213654 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.214434 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.215916 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.216404 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.216546 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.217143 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:09.217822 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:09.217989 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.218114 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.218211 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:21:09.218538 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.212401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:41809 (local address 127.0.0.1:45282)
0620 02:21:09.212639 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:09.212656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:09.212768 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:09.213314 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:09.213318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:09.213331 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:09.213519 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.213528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.214592 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.214600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.215768 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.215782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.216673 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.216678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.216855 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.216919 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:09.217944 (+ 1025us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:09.217968 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:21:09.218070 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:21:09.218239 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 02:21:09.222843 11707 raft_consensus.cc:1261] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Refusing update from remote peer 123d2b632b054c2184afacdc5c8cb209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:09.223281 11429 raft_consensus.cc:1261] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 123d2b632b054c2184afacdc5c8cb209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:09.223835 11818 consensus_queue.cc:1035] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:09.224189 11778 consensus_queue.cc:1035] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:09.237784 11793 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: TxnStatusTablet state changed. Reason: New leader 123d2b632b054c2184afacdc5c8cb209. Latest consensus state: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.238184 11793 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:09.239042 11779 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: TxnStatusTablet state changed. Reason: New leader 123d2b632b054c2184afacdc5c8cb209. Latest consensus state: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.239394 11779 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:09.241303 11778 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.241896 11778 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:21:09.242228 11787 mvcc.cc:205] Tried to move back new op lower bound from 7298752795529097216 to 7298752795366531072. Current Snapshot: MvccSnapshot[applied={T|T < 7298752795529097216}]
I20260620 02:21:09.249696 11818 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.250029 11818 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P 123d2b632b054c2184afacdc5c8cb209: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:21:09.252216 11779 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.252581 11779 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P bf1fcdfdccc34ce4a02ddb09588481bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:09.253217 11793 tablet_replica.cc:439] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "123d2b632b054c2184afacdc5c8cb209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 } } peers { permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 } } peers { permanent_uuid: "bf1fcdfdccc34ce4a02ddb09588481bb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34983 } } }
I20260620 02:21:09.253479 11793 tablet_replica.cc:442] T 38fb0247aeb04fc3aa7d59843e678013 P e883ebba2be3497da2caa1c13261f0a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:09.270421 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.258057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.9.231.67:34983)
0620 02:21:09.258296 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.258305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.264297 (+ 5992us) server_negotiation.cc:407] Connection header received
0620 02:21:09.264598 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.264602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.264692 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.264876 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:09.265432 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.266176 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.267806 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.268427 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.268566 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.269098 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:09.269720 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:09.269905 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.269967 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.270055 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:21:09.293290 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.283885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:41809 (local address 127.9.231.67:55689)
0620 02:21:09.285040 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:09.285078 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:09.285307 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:09.285691 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:09.285696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:09.285709 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:09.286346 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.286358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.288059 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.288066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.290119 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.290145 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.291610 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.291620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.291782 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.291889 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:09.292481 (+ 592us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:09.292499 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 02:21:09.292630 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:21:09.292812 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":683,"threads_started":1}
I20260620 02:21:09.293371 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.283951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:55689 (local address 127.9.231.66:41809)
0620 02:21:09.284184 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.284195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.285195 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 02:21:09.285463 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.285469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.285560 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.285670 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:09.286601 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.287867 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.290336 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.291371 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.291542 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.292137 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:09.292330 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:09.292481 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.292669 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.292758 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:21:09.311959 11563 raft_consensus.cc:1261] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Refusing update from remote peer bf1fcdfdccc34ce4a02ddb09588481bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:09.311983 11429 raft_consensus.cc:1261] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Refusing update from remote peer bf1fcdfdccc34ce4a02ddb09588481bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:09.312963 11779 consensus_queue.cc:1035] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e883ebba2be3497da2caa1c13261f0a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:09.313264 11804 consensus_queue.cc:1035] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "123d2b632b054c2184afacdc5c8cb209" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:09.396061 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.381507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:44197 (local address 127.9.231.67:34983)
0620 02:21:09.381727 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.381734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.388279 (+ 6545us) server_negotiation.cc:407] Connection header received
0620 02:21:09.388554 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.388562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.388667 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.388792 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:09.390099 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.391141 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.393537 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.394246 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.394354 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.395146 (+ 792us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:09.395317 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:09.395385 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.395614 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.395714 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 02:21:09.397017 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.381006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34983 (local address 127.9.231.66:44197)
0620 02:21:09.388144 (+ 7138us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:09.388169 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:09.388389 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:09.389309 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:09.389316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:09.389331 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:09.389856 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.389868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.391347 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.391352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.393342 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:09.393360 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.394577 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.394584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:09.394787 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.394878 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:09.395482 (+ 604us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:09.395492 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:21:09.396361 (+ 869us) client_negotiation.cc:240] Negotiation successful
0620 02:21:09.396547 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6725,"thread_start_us":6457,"threads_started":1}
I20260620 02:21:09.474514 11268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:09.474781 11268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:09.478410 11268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4992cbd9220b44dc956ce55da9c6752c: Sending DeleteTablet for 3 replicas of tablet 8a76c56de2fe49a5ad95fac9ca498022
I20260620 02:21:09.479413 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a76c56de2fe49a5ad95fac9ca498022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:09 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:39824
I20260620 02:21:09.479696 11544 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a76c56de2fe49a5ad95fac9ca498022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:09 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260620 02:21:09.480101 11682 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a76c56de2fe49a5ad95fac9ca498022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:09 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:57626
I20260620 02:21:09.488040 11839 tablet_replica.cc:330] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: stopping tablet replica
I20260620 02:21:09.488389 11839 raft_consensus.cc:2229] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:09.494065 11840 tablet_replica.cc:330] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: stopping tablet replica
I20260620 02:21:09.494434 11840 raft_consensus.cc:2229] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:09.495224 11840 raft_consensus.cc:2258] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:09.495662 11838 tablet_replica.cc:330] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: stopping tablet replica
I20260620 02:21:09.495896 11838 raft_consensus.cc:2229] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:09.496119 11838 raft_consensus.cc:2258] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:09.496737 11840 ts_tablet_manager.cc:1905] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:09.497400 11838 ts_tablet_manager.cc:1905] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:09.499164 11839 raft_consensus.cc:2258] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:09.500731 11839 ts_tablet_manager.cc:1905] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:09.504788 11840 ts_tablet_manager.cc:1918] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:09.504968 11840 log.cc:1196] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/8a76c56de2fe49a5ad95fac9ca498022
I20260620 02:21:09.505407 11838 ts_tablet_manager.cc:1918] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:09.505465 11840 ts_tablet_manager.cc:1939] T 8a76c56de2fe49a5ad95fac9ca498022 P bf1fcdfdccc34ce4a02ddb09588481bb: Deleting consensus metadata
I20260620 02:21:09.505570 11838 log.cc:1196] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/8a76c56de2fe49a5ad95fac9ca498022
I20260620 02:21:09.506106 11838 ts_tablet_manager.cc:1939] T 8a76c56de2fe49a5ad95fac9ca498022 P e883ebba2be3497da2caa1c13261f0a5: Deleting consensus metadata
I20260620 02:21:09.506722 11252 catalog_manager.cc:4841] TS bf1fcdfdccc34ce4a02ddb09588481bb (127.9.231.67:34983): tablet 8a76c56de2fe49a5ad95fac9ca498022 (table test-table [id=399381fb18104adbaac0de396db051d9]) successfully deleted
I20260620 02:21:09.507417 11254 catalog_manager.cc:4841] TS e883ebba2be3497da2caa1c13261f0a5 (127.9.231.65:35189): tablet 8a76c56de2fe49a5ad95fac9ca498022 (table test-table [id=399381fb18104adbaac0de396db051d9]) successfully deleted
I20260620 02:21:09.511011 11839 ts_tablet_manager.cc:1918] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:09.511226 11839 log.cc:1196] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/8a76c56de2fe49a5ad95fac9ca498022
I20260620 02:21:09.511955 11839 ts_tablet_manager.cc:1939] T 8a76c56de2fe49a5ad95fac9ca498022 P 123d2b632b054c2184afacdc5c8cb209: Deleting consensus metadata
I20260620 02:21:09.513494 11253 catalog_manager.cc:4841] TS 123d2b632b054c2184afacdc5c8cb209 (127.9.231.66:41809): tablet 8a76c56de2fe49a5ad95fac9ca498022 (table test-table [id=399381fb18104adbaac0de396db051d9]) successfully deleted
Jun 20 02:21:09 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:09.528538 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.493343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.9.231.65:35189)
0620 02:21:09.493613 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.493619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.508609 (+ 14990us) server_negotiation.cc:407] Connection header received
0620 02:21:09.511605 (+ 2996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.511615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.511719 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.511804 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:09.514423 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.515319 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.516688 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.517291 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.517430 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.521862 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:09.521882 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:09.521904 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:09.521958 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:09.525247 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.526092 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.526100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.526103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.526259 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.526729 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.526734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.526739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.527211 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:09.527578 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.527884 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.528122 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 02:21:09.529376 11470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39846
I20260620 02:21:09.551666 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.533019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.9.231.126:44385)
0620 02:21:09.536803 (+ 3784us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.536809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.537163 (+ 354us) server_negotiation.cc:407] Connection header received
0620 02:21:09.537237 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.537241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.537305 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.537442 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:09.539097 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.539871 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.541294 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.541790 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.541950 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.544922 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:09.544935 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:09.544939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:09.544987 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:09.548288 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.549002 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.549007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.549011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.549150 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.549554 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.549559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.549562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.549880 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:09.550327 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.550769 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.551111 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3631,"thread_start_us":3413,"threads_started":1}
W20260620 02:21:09.552289 11268 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52682
I20260620 02:21:09.554574 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11339
I20260620 02:21:09.576660 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.556078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.9.231.65:35189)
0620 02:21:09.557352 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.557361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.563658 (+ 6297us) server_negotiation.cc:407] Connection header received
0620 02:21:09.563892 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.563896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.563966 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.564077 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:09.565432 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.566337 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.567902 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.568328 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.568468 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.571149 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:09.571155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:09.571158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:09.571204 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:09.573927 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.574627 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.574633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.574637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.574782 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.575234 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.575239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.575243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.575521 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:09.575787 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.576092 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.576324 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063}
W20260620 02:21:09.577476 11470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39858
W20260620 02:21:09.578207 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11339: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:09.578462 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 11339
I20260620 02:21:09.604065 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11479
Jun 20 02:21:09 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:21:09.621387 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.605075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.9.231.66:41809)
0620 02:21:09.605282 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.605290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.605647 (+ 357us) server_negotiation.cc:407] Connection header received
0620 02:21:09.605872 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.605877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.605971 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.606118 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:09.607796 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.608711 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.610025 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.610494 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.610649 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.614857 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:09.614871 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:09.614887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:09.614932 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:09.618271 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.619140 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.619148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.619151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.619296 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.619697 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.619703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.619706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.620079 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:09.620440 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.620659 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.620886 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 02:21:09.622112 11609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45292
W20260620 02:21:09.622792 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:09.622975 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 11479
I20260620 02:21:09.645797 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11618
Jun 20 02:21:09 dist-test-slave-9vg1 krb5kdc[11223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:21:09.665452 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:09.646765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.9.231.67:34983)
0620 02:21:09.646958 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:09.646965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:09.646988 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:21:09.647224 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:09.647230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:09.647336 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:09.647466 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:09.648829 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.649959 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.651401 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:09.652122 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:09.652272 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:09.657124 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:09.657152 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:09.657175 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:09.657248 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:09.661632 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.662624 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.662635 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.662640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.662805 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:09.663416 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:09.663422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:09.663426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:09.663925 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:09.664262 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:09.664604 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:21:09.664851 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 02:21:09.666247 11748 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57646
W20260620 02:21:09.667022 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11618: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:09.667228 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 11618
I20260620 02:21:09.683375 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11236
I20260620 02:21:09.684676 11308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:09.806504 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 11236
2026-06-20T02:21:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4934 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:09 dist-test-slave-9vg1 krb5kdc[11850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:09 dist-test-slave-9vg1 krb5kdc[11850](info): set up 2 sockets
Jun 20 02:21:09 dist-test-slave-9vg1 krb5kdc[11850](info): commencing operation
krb5kdc: starting...
W20260620 02:21:10.920595 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:11 dist-test-slave-9vg1 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:11.128041 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44557
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:44557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:11.470799 11863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:11.471204 11863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:11.471364 11863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:11.478811 11863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:11.478928 11863 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:21:11.479001 11863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:11.479113 11863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:11.479195 11863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:11.479275 11863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:11.489542 11863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:44557
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:11.490957 11863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:11.492455 11863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:11.505000 11871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:11.505144 11868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:11.505102 11869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:11.506397 11863 server_base.cc:1034] running on GCE node
I20260620 02:21:11.507344 11863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:11.508987 11863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:11.510262 11863 hybrid_clock.cc:648] HybridClock initialized: now 1781922071510182 us; error 67 us; skew 500 ppm
I20260620 02:21:11.510670 11863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:11 dist-test-slave-9vg1 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922071, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:11.514372 11863 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:11.516144 11863 webserver.cc:460] Webserver started at http://127.9.231.126:33913/ using document root <none> and password file <none>
I20260620 02:21:11.516947 11863 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:11.517087 11863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:11.517426 11863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:11.520607 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "267a55f972564abebaef8736f68013cd"
format_stamp: "Formatted at 2026-06-20 02:21:11 on dist-test-slave-9vg1"
server_key: "94ccf7b4cc025705dd82be18c45c0555"
server_key_iv: "38dfe72b6167ec0e88aace193cb5d154"
server_key_version: "encryptionkey@0"
I20260620 02:21:11.521423 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "267a55f972564abebaef8736f68013cd"
format_stamp: "Formatted at 2026-06-20 02:21:11 on dist-test-slave-9vg1"
server_key: "94ccf7b4cc025705dd82be18c45c0555"
server_key_iv: "38dfe72b6167ec0e88aace193cb5d154"
server_key_version: "encryptionkey@0"
I20260620 02:21:11.527186 11863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 02:21:11.531118 11878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:11.532706 11863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:21:11.533020 11863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "267a55f972564abebaef8736f68013cd"
format_stamp: "Formatted at 2026-06-20 02:21:11 on dist-test-slave-9vg1"
server_key: "94ccf7b4cc025705dd82be18c45c0555"
server_key_iv: "38dfe72b6167ec0e88aace193cb5d154"
server_key_version: "encryptionkey@0"
I20260620 02:21:11.533272 11863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:11.547199 11863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:11.551965 11863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:11.552274 11863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:11.575049 11863 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:44557
I20260620 02:21:11.575088 11939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:44557 every 8 connection(s)
I20260620 02:21:11.576562 11863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:11.578313 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 11863
I20260620 02:21:11.578539 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:11.579393 10141 external_mini_cluster.cc:1428] Setting key bee6dd9ee6287d2ff7a89432ee762f7f
I20260620 02:21:11.581646 11940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:21:11 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:11.596825 11940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Bootstrap starting.
I20260620 02:21:11.600389 11940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:11.601517 11940 log.cc:824] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:11.604736 11940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: No bootstrap required, opened a new log
I20260620 02:21:11.604877 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:11.581645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.9.231.126:44557)
0620 02:21:11.582504 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:11.582517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:11.582559 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:21:11.584238 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:11.584261 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:11.584682 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:11.585097 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:11.586994 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:11.588754 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:11.590656 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:11.591647 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:11.591800 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:11.597123 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:11.597133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:11.597154 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:11.597208 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:11.600518 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:11.601347 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:11.601355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:11.601362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:11.601535 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:11.601946 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:11.601951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:11.601954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:11.602533 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:11.602849 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:11.603354 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 02:21:11.603782 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":418,"threads_started":1}
I20260620 02:21:11.613669 11940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267a55f972564abebaef8736f68013cd" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44557 } }
I20260620 02:21:11.614130 11940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:11.614243 11940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267a55f972564abebaef8736f68013cd, State: Initialized, Role: FOLLOWER
I20260620 02:21:11.614930 11940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267a55f972564abebaef8736f68013cd" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44557 } }
I20260620 02:21:11.615270 11940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:11.615464 11940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:11.615653 11940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:11.618181 11940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267a55f972564abebaef8736f68013cd" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44557 } }
I20260620 02:21:11.618763 11940 leader_election.cc:302] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 267a55f972564abebaef8736f68013cd; no voters:
I20260620 02:21:11.619522 11940 leader_election.cc:288] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:11.619709 11945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:11.620496 11945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [term 1 LEADER]: Becoming Leader. State: Replica: 267a55f972564abebaef8736f68013cd, State: Running, Role: LEADER
I20260620 02:21:11.621078 11945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267a55f972564abebaef8736f68013cd" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44557 } }
I20260620 02:21:11.621748 11940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:11.626130 11946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "267a55f972564abebaef8736f68013cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267a55f972564abebaef8736f68013cd" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44557 } } }
I20260620 02:21:11.626545 11946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:11.626957 11947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 267a55f972564abebaef8736f68013cd. Latest consensus state: current_term: 1 leader_uuid: "267a55f972564abebaef8736f68013cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267a55f972564abebaef8736f68013cd" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44557 } } }
I20260620 02:21:11.627298 11947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:11.627606 11956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:11.634867 11956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:11.645068 11956 catalog_manager.cc:1347] Generated new cluster ID: 98abbd23fab947df96824cd69755c23f
I20260620 02:21:11.645203 11956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:11.657614 11956 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:11.659026 11956 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:11.670730 11956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Generated new TSK 0
I20260620 02:21:11.671702 11956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:11.727736 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44557
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:12.083722 11966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:12.084100 11966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:12.084290 11966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:12.092042 11966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:12.092232 11966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:12.092422 11966 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:21:12.092516 11966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:21:12.092659 11966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:12.103190 11966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:12.104615 11966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:12.106024 11966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:12.119971 11971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:12.120016 11972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:12.119887 11974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:12.124783 11966 server_base.cc:1034] running on GCE node
I20260620 02:21:12.125676 11966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:12.126868 11966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:12.128123 11966 hybrid_clock.cc:648] HybridClock initialized: now 1781922072128052 us; error 70 us; skew 500 ppm
I20260620 02:21:12.128562 11966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:12 dist-test-slave-9vg1 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:12.132179 11966 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:12.133988 11966 webserver.cc:460] Webserver started at http://127.9.231.65:38273/ using document root <none> and password file <none>
I20260620 02:21:12.134797 11966 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:12.134966 11966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:12.135363 11966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:12.138912 11966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac"
format_stamp: "Formatted at 2026-06-20 02:21:12 on dist-test-slave-9vg1"
server_key: "93eaef29ca214a660b850dfe6de62090"
server_key_iv: "88eeab506910e108cc3c206827f2100b"
server_key_version: "encryptionkey@0"
I20260620 02:21:12.139791 11966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac"
format_stamp: "Formatted at 2026-06-20 02:21:12 on dist-test-slave-9vg1"
server_key: "93eaef29ca214a660b850dfe6de62090"
server_key_iv: "88eeab506910e108cc3c206827f2100b"
server_key_version: "encryptionkey@0"
I20260620 02:21:12.145675 11966 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:21:12.149722 11981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:12.151409 11966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:12.151615 11966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac"
format_stamp: "Formatted at 2026-06-20 02:21:12 on dist-test-slave-9vg1"
server_key: "93eaef29ca214a660b850dfe6de62090"
server_key_iv: "88eeab506910e108cc3c206827f2100b"
server_key_version: "encryptionkey@0"
I20260620 02:21:12.151835 11966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:12.173826 11966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:12.178601 11966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:12.178921 11966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:12.181968 11966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:12.182075 11966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:12.182255 11966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:12.182368 11966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:21:12 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:12.208817 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.182459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:60825 (local address 127.9.231.126:44557)
0620 02:21:12.183035 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:12.183042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:12.184343 (+ 1301us) server_negotiation.cc:407] Connection header received
0620 02:21:12.187909 (+ 3566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:12.187914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:12.188018 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:12.188138 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:12.190773 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.192429 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.195043 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.195676 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.195839 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.201433 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:12.201440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:12.201443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:12.201482 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:12.204799 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.205963 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.205967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.205970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.206092 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.206595 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.206602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.206609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.206990 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:12.207165 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:12.208186 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0620 02:21:12.208425 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":252,"threads_started":1}
I20260620 02:21:12.210165 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.183160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44557 (local address 127.9.231.65:60825)
0620 02:21:12.184120 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:12.184179 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:12.187487 (+ 3308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:12.188555 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:12.188572 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:12.189536 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:12.190469 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.190496 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.192727 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.192736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.194847 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.194865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.195967 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.195975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.196390 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.197407 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:12.197455 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:12.201143 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:12.205066 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.205079 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.205107 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.205784 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.206221 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.206226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.206230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.206457 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.207156 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:12.207164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:12.207758 (+ 594us) client_negotiation.cc:769] Sending connection context
0620 02:21:12.208089 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 02:21:12.208682 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":340,"threads_started":1}
I20260620 02:21:12.235162 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.217170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44557 (local address 127.9.231.65:34895)
0620 02:21:12.218068 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:12.218095 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:12.218341 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:12.218702 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:12.218706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:12.219278 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:12.219849 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.219866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.221251 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.221261 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.223831 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.223857 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.224782 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.224793 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.225088 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.225940 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:12.225958 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:12.226830 (+ 872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:12.231644 (+ 4814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.231657 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.231665 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.232565 (+ 900us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.233056 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.233062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.233065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.233290 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.234168 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:12.234172 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:12.234339 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 02:21:12.234470 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:21:12.234754 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":765,"threads_started":1}
I20260620 02:21:12.235616 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.217683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:34895 (local address 127.9.231.126:44557)
0620 02:21:12.217931 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:12.217942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:12.218201 (+ 259us) server_negotiation.cc:407] Connection header received
0620 02:21:12.218481 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:12.218493 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:12.218570 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:12.218678 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:12.220016 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.221070 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.224039 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.224468 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.224576 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.227117 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:12.227130 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:12.227139 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:12.227201 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:12.231428 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.232741 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.232749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.232754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.232919 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.233446 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.233456 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.233464 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.234017 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:12.234279 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:12.234648 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 02:21:12.235087 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:21:12.242743 11966 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:39505
I20260620 02:21:12.242774 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:39505 every 8 connection(s)
I20260620 02:21:12.244174 11966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:12.246636 12102 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44557
I20260620 02:21:12.246922 12102 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:12.247748 12102 heartbeater.cc:511] Master 127.9.231.126:44557 requested a full tablet report, sending...
I20260620 02:21:12.250048 11895 ts_manager.cc:194] Registered new tserver with Master: 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505)
I20260620 02:21:12.251802 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 11966
I20260620 02:21:12.251806 11895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:60825
I20260620 02:21:12.252034 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:12.252772 10141 external_mini_cluster.cc:1428] Setting key b9c0c503e00b604c21af27d447cc0aba
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:12.299408 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44557
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:12.646706 12106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:12.647163 12106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:12.647302 12106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:12.655793 12106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:12.655959 12106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:12.656092 12106 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:21:12.656158 12106 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:21:12.656237 12106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:12.667178 12106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:12.668583 12106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:12.670027 12106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:12.684347 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:12.684484 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:12.684393 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:12.685276 12106 server_base.cc:1034] running on GCE node
I20260620 02:21:12.685977 12106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:12.687141 12106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:12.688395 12106 hybrid_clock.cc:648] HybridClock initialized: now 1781922072688339 us; error 53 us; skew 500 ppm
I20260620 02:21:12.688761 12106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:12 dist-test-slave-9vg1 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:12.692236 12106 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:12.693965 12106 webserver.cc:460] Webserver started at http://127.9.231.66:36009/ using document root <none> and password file <none>
I20260620 02:21:12.694710 12106 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:12.694862 12106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:12.695287 12106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:12.698377 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "075c5a223b0845479bb74bf8b4e2d570"
format_stamp: "Formatted at 2026-06-20 02:21:12 on dist-test-slave-9vg1"
server_key: "39f7d70d53ea519e6e38b35a8c52123b"
server_key_iv: "044f51d2a643d56fb8175b95bcb96eb3"
server_key_version: "encryptionkey@0"
I20260620 02:21:12.699275 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "075c5a223b0845479bb74bf8b4e2d570"
format_stamp: "Formatted at 2026-06-20 02:21:12 on dist-test-slave-9vg1"
server_key: "39f7d70d53ea519e6e38b35a8c52123b"
server_key_iv: "044f51d2a643d56fb8175b95bcb96eb3"
server_key_version: "encryptionkey@0"
I20260620 02:21:12.704721 12106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:21:12.708612 12121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:12.710069 12106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:12.710263 12106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "075c5a223b0845479bb74bf8b4e2d570"
format_stamp: "Formatted at 2026-06-20 02:21:12 on dist-test-slave-9vg1"
server_key: "39f7d70d53ea519e6e38b35a8c52123b"
server_key_iv: "044f51d2a643d56fb8175b95bcb96eb3"
server_key_version: "encryptionkey@0"
I20260620 02:21:12.710479 12106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:12.724344 12106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:12.729413 12106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:12.729749 12106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:12.732707 12106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:12.732841 12106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:12.732956 12106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:12.733047 12106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:21:12 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:12.758559 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.733615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:33487 (local address 127.9.231.126:44557)
0620 02:21:12.733828 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:12.733833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:12.735597 (+ 1764us) server_negotiation.cc:407] Connection header received
0620 02:21:12.737988 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:12.737994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:12.738109 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:12.738250 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:12.740802 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.741872 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.745052 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.745537 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.745704 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.751492 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:12.751499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:12.751502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:12.751539 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:12.754411 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.755366 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.755372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.755375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.755580 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.756146 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.756151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.756154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.756681 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:12.756914 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:12.757961 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0620 02:21:12.758194 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:21:12.759711 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.734330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44557 (local address 127.9.231.66:33487)
0620 02:21:12.735355 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:12.735421 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:12.737574 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:12.738553 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:12.738570 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:12.739545 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:12.740577 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.740614 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.742104 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.742110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.744825 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.744848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.746025 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.746039 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.746466 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.747565 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:12.747635 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:12.751195 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:12.754579 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.754589 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.754609 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.755216 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.755702 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.755707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.755711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.755957 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.756840 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:12.756849 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:12.757323 (+ 474us) client_negotiation.cc:769] Sending connection context
0620 02:21:12.757833 (+ 510us) client_negotiation.cc:240] Negotiation successful
0620 02:21:12.758442 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":417,"threads_started":1}
I20260620 02:21:12.787423 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.767364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46319 (local address 127.9.231.126:44557)
0620 02:21:12.767580 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:12.767592 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:12.771718 (+ 4126us) server_negotiation.cc:407] Connection header received
0620 02:21:12.771936 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:12.771940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:12.772027 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:12.772186 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:12.773836 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.774573 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.777032 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.777700 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.777847 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.780775 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:12.780781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:12.780784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:12.780828 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:12.784494 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.785262 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.785275 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.785277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.785413 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:12.785792 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:12.785801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:12.785803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:12.786149 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:12.786310 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:12.786731 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 02:21:12.786995 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:21:12.787658 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:12.767389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44557 (local address 127.9.231.66:46319)
0620 02:21:12.771521 (+ 4132us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:12.771551 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:12.771797 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:12.772227 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:12.772232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:12.773099 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:12.773603 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.773624 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.774891 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.774900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.776816 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:12.776832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:12.777948 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:12.777954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:12.778105 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:12.779156 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:12.779182 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:12.780546 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:12.784664 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.784670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.784675 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.785112 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.785530 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:12.785535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:12.785538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:12.785670 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:12.786274 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:12.786280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:12.786481 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 02:21:12.786626 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 02:21:12.786995 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3879,"thread_start_us":3373,"threads_started":1}
I20260620 02:21:12.794965 12106 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:36167
I20260620 02:21:12.795011 12240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:36167 every 8 connection(s)
I20260620 02:21:12.796309 12106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:12.798779 12241 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44557
I20260620 02:21:12.799089 12241 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:12.800078 12241 heartbeater.cc:511] Master 127.9.231.126:44557 requested a full tablet report, sending...
I20260620 02:21:12.801625 11895 ts_manager.cc:194] Registered new tserver with Master: 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167)
I20260620 02:21:12.802424 11895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:33487
I20260620 02:21:12.803378 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 12106
I20260620 02:21:12.803634 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:12.804083 10141 external_mini_cluster.cc:1428] Setting key 13ddfd2779c07bb444129970a6783811
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:12.850427 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44557
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:13.210654 12245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:13.211019 12245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:13.211225 12245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:13.219449 12245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:13.219628 12245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:13.219774 12245 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:21:13.219863 12245 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:21:13.219997 12245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:13.230628 12245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:13.232231 12245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:13.233842 12245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:13.249702 12253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:13.249769 12251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:13.250348 12250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:13.251123 12245 server_base.cc:1034] running on GCE node
I20260620 02:21:13.251806 12245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:13.252884 12245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:13.254145 12245 hybrid_clock.cc:648] HybridClock initialized: now 1781922073254094 us; error 64 us; skew 500 ppm
I20260620 02:21:13.254493 12245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:13.255319 12102 heartbeater.cc:503] Master 127.9.231.126:44557 was elected leader, sending a full tablet report...
Jun 20 02:21:13 dist-test-slave-9vg1 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922073, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:13.258072 12245 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:13.259682 12245 webserver.cc:460] Webserver started at http://127.9.231.67:35153/ using document root <none> and password file <none>
I20260620 02:21:13.260345 12245 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:13.260448 12245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:13.260710 12245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:13.263700 12245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "d17065c9ddcf44de90aba68d08d779cb"
format_stamp: "Formatted at 2026-06-20 02:21:13 on dist-test-slave-9vg1"
server_key: "8274fc01561914d0ca017c92b3b33e45"
server_key_iv: "ad6fafe7c3425cbf455523ecf344f4ba"
server_key_version: "encryptionkey@0"
I20260620 02:21:13.264442 12245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "d17065c9ddcf44de90aba68d08d779cb"
format_stamp: "Formatted at 2026-06-20 02:21:13 on dist-test-slave-9vg1"
server_key: "8274fc01561914d0ca017c92b3b33e45"
server_key_iv: "ad6fafe7c3425cbf455523ecf344f4ba"
server_key_version: "encryptionkey@0"
I20260620 02:21:13.269974 12245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:21:13.273936 12260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:13.275470 12245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:13.275637 12245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "d17065c9ddcf44de90aba68d08d779cb"
format_stamp: "Formatted at 2026-06-20 02:21:13 on dist-test-slave-9vg1"
server_key: "8274fc01561914d0ca017c92b3b33e45"
server_key_iv: "ad6fafe7c3425cbf455523ecf344f4ba"
server_key_version: "encryptionkey@0"
I20260620 02:21:13.275861 12245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:13.294106 12245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:13.298914 12245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:13.299240 12245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:13.302130 12245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:13.302225 12245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:13.302410 12245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:13.302520 12245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:21:13 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922073, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:13.329699 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.302693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:54815 (local address 127.9.231.126:44557)
0620 02:21:13.305769 (+ 3076us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.305774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.305794 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:21:13.308202 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.308209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.308326 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.308473 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:13.311136 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.312184 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.314941 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.315893 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.316086 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.321162 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:13.321174 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:13.321182 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:13.321243 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:13.325594 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.326618 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.326627 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.326633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.326831 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.327673 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.327680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.327684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.328004 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:13.328175 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.329057 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.329300 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2969,"thread_start_us":2851,"threads_started":1}
I20260620 02:21:13.330803 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.303562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44557 (local address 127.9.231.67:54815)
0620 02:21:13.304673 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.304731 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.307779 (+ 3048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.308844 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.308905 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.309770 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:13.310886 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.310910 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.312400 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.312406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.314668 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.314692 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.316214 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.316223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.316605 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.317426 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:13.317477 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:13.320795 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:13.325809 (+ 5014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:13.325820 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:13.325849 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:13.326457 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:13.326987 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:13.326995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:13.327001 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:13.327452 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:13.328171 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:13.328182 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:13.328656 (+ 474us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.328939 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.329522 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":447,"threads_started":1}
I20260620 02:21:13.354516 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.337655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:39411 (local address 127.9.231.126:44557)
0620 02:21:13.337836 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.337840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.338542 (+ 702us) server_negotiation.cc:407] Connection header received
0620 02:21:13.338697 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.338706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.338833 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.338969 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:13.340060 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.341519 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.343001 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.343867 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.344027 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.347395 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:13.347402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:13.347406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:13.347443 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:13.351038 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.352114 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.352118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.352120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.352247 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.352707 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.352710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.352712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.352998 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:13.353110 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.353775 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.354039 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:21:13.354655 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.337776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44557 (local address 127.9.231.67:39411)
0620 02:21:13.338443 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.338459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.338582 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.338992 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.338995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.339595 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:13.339895 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.339907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.341721 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.341726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.342843 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.342856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.344352 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.344362 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.344656 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.345594 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:13.345622 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:13.347143 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:13.351239 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:13.351248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:13.351256 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:13.351975 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:13.352363 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:13.352371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:13.352377 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:13.352597 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:13.353115 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:13.353123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:13.353400 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.353584 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.354074 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":346,"threads_started":1}
I20260620 02:21:13.363196 12245 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:36913
I20260620 02:21:13.363217 12380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:36913 every 8 connection(s)
I20260620 02:21:13.364476 12245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:13.366832 12381 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44557
I20260620 02:21:13.367174 12381 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:13.368009 12381 heartbeater.cc:511] Master 127.9.231.126:44557 requested a full tablet report, sending...
I20260620 02:21:13.369582 11895 ts_manager.cc:194] Registered new tserver with Master: d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913)
I20260620 02:21:13.370482 11895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:54815
I20260620 02:21:13.374974 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 12245
I20260620 02:21:13.375226 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:13.375816 10141 external_mini_cluster.cc:1428] Setting key a85ed62b7c333efae02b56b89999146f
I20260620 02:21:13.379679 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:21:13 dist-test-slave-9vg1 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:21:13 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:13.431469 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.415941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.9.231.126:44557)
0620 02:21:13.416123 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.416127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.416503 (+ 376us) server_negotiation.cc:407] Connection header received
0620 02:21:13.416656 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.416660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.416742 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.416834 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:13.418429 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.419245 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.420669 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.421091 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.421191 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.425756 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:13.425762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:13.425765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:13.425824 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:13.428727 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.429480 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.429485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.429487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.429640 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.430017 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.430021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.430023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.430316 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:13.430580 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.430801 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.430967 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:21:13.435902 11895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:13.443221 11895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:13.479171 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.457396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.9.231.66:36167)
0620 02:21:13.468861 (+ 11465us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.468872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.469630 (+ 758us) server_negotiation.cc:407] Connection header received
0620 02:21:13.470096 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.470106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.470392 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.470562 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.472661 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.473919 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.476431 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.477858 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.478039 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.478275 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.478560 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.478765 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11285,"thread_start_us":11155,"threads_started":1}
I20260620 02:21:13.480183 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.457228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.9.231.67:36913)
0620 02:21:13.468146 (+ 10918us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.468161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.469630 (+ 1469us) server_negotiation.cc:407] Connection header received
0620 02:21:13.470096 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.470106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.470387 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.470573 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.471898 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.473618 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.476695 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.478086 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.478202 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.478399 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.479612 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.479785 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10630,"thread_start_us":10942,"threads_started":1}
I20260620 02:21:13.480250 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.470255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.9.231.65:39505)
0620 02:21:13.472533 (+ 2278us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.472543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.472576 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:21:13.472675 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.472683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.472945 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.473133 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.473749 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.475345 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.477437 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.478641 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.478774 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.478966 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.479845 (+ 879us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.480043 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2110,"thread_start_us":1963,"threads_started":1}
I20260620 02:21:13.480917 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.470040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:39505 (local address 127.0.0.1:53512)
0620 02:21:13.471806 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.471825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.471928 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.473154 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.473159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.473188 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.473555 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.473567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.475536 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.475542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.477286 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.477298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.479174 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.479181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.479477 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.479509 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.479593 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.479698 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1591,"thread_start_us":1431,"threads_started":1}
I20260620 02:21:13.481093 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.457177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:36167 (local address 127.0.0.1:59498)
0620 02:21:13.469402 (+ 12225us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.469490 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.469821 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.470912 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.470921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.470959 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.472473 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.472488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.474047 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.474055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.476262 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.476274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.478074 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.478081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.478268 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.478324 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.479437 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.479624 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11993,"mutex_wait_us":957,"spinlock_wait_cycles":9088,"thread_start_us":11848,"threads_started":1}
I20260620 02:21:13.481114 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.456756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:36913 (local address 127.0.0.1:42018)
0620 02:21:13.469402 (+ 12646us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.469490 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.469821 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.470912 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.470921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.470962 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.471638 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.471653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.473829 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.473837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.476510 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.476531 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.478539 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.478545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.478746 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.478787 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.478886 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.479131 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12332,"mutex_wait_us":116,"spinlock_wait_cycles":22144,"thread_start_us":12175,"threads_started":1}
I20260620 02:21:13.484539 12171 tablet_service.cc:1467] Processing CreateTablet for tablet 847e6c74f4474dc69b63eb17e7513523 (DEFAULT_TABLE table=test-table [id=7d460e50aa4b4fa3b252942fe35cd432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:13.484722 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 847e6c74f4474dc69b63eb17e7513523 (DEFAULT_TABLE table=test-table [id=7d460e50aa4b4fa3b252942fe35cd432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:13.485718 12171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847e6c74f4474dc69b63eb17e7513523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:13.485371 12311 tablet_service.cc:1467] Processing CreateTablet for tablet 847e6c74f4474dc69b63eb17e7513523 (DEFAULT_TABLE table=test-table [id=7d460e50aa4b4fa3b252942fe35cd432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:13.486073 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847e6c74f4474dc69b63eb17e7513523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:13.486662 12311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847e6c74f4474dc69b63eb17e7513523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:13.497545 12400 tablet_bootstrap.cc:492] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Bootstrap starting.
I20260620 02:21:13.501577 12400 tablet_bootstrap.cc:654] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:13.501860 12401 tablet_bootstrap.cc:492] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Bootstrap starting.
I20260620 02:21:13.502952 12400 log.cc:824] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:13.503537 12402 tablet_bootstrap.cc:492] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Bootstrap starting.
I20260620 02:21:13.506624 12402 tablet_bootstrap.cc:654] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:13.506656 12401 tablet_bootstrap.cc:654] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:13.507912 12402 log.cc:824] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:13.508422 12401 log.cc:824] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:13.510701 12402 tablet_bootstrap.cc:492] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: No bootstrap required, opened a new log
I20260620 02:21:13.511032 12402 ts_tablet_manager.cc:1397] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 02:21:13.511672 12400 tablet_bootstrap.cc:492] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: No bootstrap required, opened a new log
I20260620 02:21:13.511947 12400 ts_tablet_manager.cc:1397] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260620 02:21:13.513391 12401 tablet_bootstrap.cc:492] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: No bootstrap required, opened a new log
I20260620 02:21:13.513655 12401 ts_tablet_manager.cc:1397] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 02:21:13.519671 12402 raft_consensus.cc:348] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.520095 12402 raft_consensus.cc:374] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:13.520190 12402 raft_consensus.cc:729] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d35f6d913cb4b3ea0f5b66b9d2902ac, State: Initialized, Role: FOLLOWER
I20260620 02:21:13.520838 12402 consensus_queue.cc:260] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.522751 12402 ts_tablet_manager.cc:1428] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260620 02:21:13.523877 12400 raft_consensus.cc:348] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.524365 12400 raft_consensus.cc:374] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:13.524482 12400 raft_consensus.cc:729] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075c5a223b0845479bb74bf8b4e2d570, State: Initialized, Role: FOLLOWER
I20260620 02:21:13.524462 12401 raft_consensus.cc:348] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.524961 12401 raft_consensus.cc:374] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:13.525090 12401 raft_consensus.cc:729] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17065c9ddcf44de90aba68d08d779cb, State: Initialized, Role: FOLLOWER
I20260620 02:21:13.525329 12400 consensus_queue.cc:260] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.525734 12401 consensus_queue.cc:260] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.528437 12241 heartbeater.cc:503] Master 127.9.231.126:44557 was elected leader, sending a full tablet report...
I20260620 02:21:13.529256 12400 ts_tablet_manager.cc:1428] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260620 02:21:13.530200 12381 heartbeater.cc:503] Master 127.9.231.126:44557 was elected leader, sending a full tablet report...
I20260620 02:21:13.531019 12401 ts_tablet_manager.cc:1428] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
W20260620 02:21:13.548375 12242 tablet.cc:2369] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:21:13.616277 12382 tablet.cc:2369] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:21:13.746927 12103 tablet.cc:2369] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:13.781145 12407 raft_consensus.cc:482] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:13.781445 12407 raft_consensus.cc:504] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.783314 12407 leader_election.cc:288] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913), 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505)
I20260620 02:21:13.790798 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.783665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:35189 (local address 127.9.231.67:36913)
0620 02:21:13.783901 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.783910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.784531 (+ 621us) server_negotiation.cc:407] Connection header received
0620 02:21:13.784769 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.784779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.784884 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.784964 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.785797 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.786693 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.788985 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.789752 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.789953 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.790036 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.790324 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.790457 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:21:13.790845 12409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.783644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:36913 (local address 127.9.231.66:35189)
0620 02:21:13.784403 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.784429 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.784577 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.785255 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.785261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.785292 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.785653 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.785662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.786831 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.786838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.788823 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.788839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.789991 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.789997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.790162 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.790187 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.790297 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.790450 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":454,"threads_started":1}
I20260620 02:21:13.792138 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847e6c74f4474dc69b63eb17e7513523" candidate_uuid: "075c5a223b0845479bb74bf8b4e2d570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17065c9ddcf44de90aba68d08d779cb" is_pre_election: true
I20260620 02:21:13.792685 12336 raft_consensus.cc:2454] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075c5a223b0845479bb74bf8b4e2d570 in term 0.
I20260620 02:21:13.793378 12122 leader_election.cc:302] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:13.793836 12407 raft_consensus.cc:2781] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:13.793999 12407 raft_consensus.cc:482] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:13.794137 12407 raft_consensus.cc:3037] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:13.794936 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.784810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:41189 (local address 127.9.231.65:39505)
0620 02:21:13.785054 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.785062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.785800 (+ 738us) server_negotiation.cc:407] Connection header received
0620 02:21:13.786089 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.786095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.786181 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.786276 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.787132 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.788283 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.791338 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.793130 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.793315 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.793440 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.794256 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.794589 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:21:13.795984 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.784578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:39505 (local address 127.9.231.66:41189)
0620 02:21:13.785670 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.785692 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.785913 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.786328 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.786336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.786381 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.786877 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.786897 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.788441 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.788449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.791081 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.791104 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.793705 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.793722 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.793995 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.794049 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.795258 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.795487 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":655,"threads_started":1}
I20260620 02:21:13.797691 12407 raft_consensus.cc:504] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.797655 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847e6c74f4474dc69b63eb17e7513523" candidate_uuid: "075c5a223b0845479bb74bf8b4e2d570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" is_pre_election: true
I20260620 02:21:13.798395 12407 leader_election.cc:288] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [CANDIDATE]: Term 1 election: Requested vote from peers d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913), 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505)
I20260620 02:21:13.798419 12056 raft_consensus.cc:2454] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075c5a223b0845479bb74bf8b4e2d570 in term 0.
I20260620 02:21:13.799278 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847e6c74f4474dc69b63eb17e7513523" candidate_uuid: "075c5a223b0845479bb74bf8b4e2d570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac"
I20260620 02:21:13.799474 12056 raft_consensus.cc:3037] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:13.799463 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847e6c74f4474dc69b63eb17e7513523" candidate_uuid: "075c5a223b0845479bb74bf8b4e2d570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17065c9ddcf44de90aba68d08d779cb"
I20260620 02:21:13.799738 12336 raft_consensus.cc:3037] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:13.802915 12056 raft_consensus.cc:2454] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075c5a223b0845479bb74bf8b4e2d570 in term 1.
I20260620 02:21:13.803577 12336 raft_consensus.cc:2454] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075c5a223b0845479bb74bf8b4e2d570 in term 1.
I20260620 02:21:13.803910 12122 leader_election.cc:302] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, 7d35f6d913cb4b3ea0f5b66b9d2902ac; no voters:
I20260620 02:21:13.804273 12407 raft_consensus.cc:2781] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:13.804821 12407 raft_consensus.cc:686] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 LEADER]: Becoming Leader. State: Replica: 075c5a223b0845479bb74bf8b4e2d570, State: Running, Role: LEADER
I20260620 02:21:13.805413 12407 consensus_queue.cc:237] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } }
I20260620 02:21:13.810456 11894 catalog_manager.cc:5495] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "075c5a223b0845479bb74bf8b4e2d570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } health_report { overall_health: HEALTHY } } }
Jun 20 02:21:13 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922071, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:13.896545 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.881950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.9.231.126:44557)
0620 02:21:13.882132 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.882136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.882715 (+ 579us) server_negotiation.cc:407] Connection header received
0620 02:21:13.882933 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.882938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.882998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.883137 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:13.884303 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.885122 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.886374 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.886806 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.886952 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.890575 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:13.890581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:13.890584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:13.890619 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:13.893582 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.894364 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.894371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.894375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.894517 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:13.894902 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:13.894908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:13.894911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:13.895245 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:13.895396 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.895828 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.896190 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":60}
I20260620 02:21:13.896551 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.881977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44557 (local address 127.0.0.1:34122)
0620 02:21:13.882604 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.882618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.882801 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.883159 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.883163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.883885 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:13.884143 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.884152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.885262 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.885269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.886219 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.886230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.886994 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.887000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.887182 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.887865 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:13.887886 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:13.890318 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:13.893761 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:13.893779 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:13.893796 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:13.894229 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:13.894629 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:13.894635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:13.894638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:13.894768 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:13.895371 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:13.895379 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:13.895562 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.895734 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.896051 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260620 02:21:13.902525 11894 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:34122:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:21:13.903852 11894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:13.911201 12171 tablet_service.cc:1467] Processing CreateTablet for tablet addc32413df14a3082658f77f79271d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fef1945ec77d4e5787d64886b246ab52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:21:13.911182 12032 tablet_service.cc:1467] Processing CreateTablet for tablet addc32413df14a3082658f77f79271d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fef1945ec77d4e5787d64886b246ab52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:21:13.911441 12311 tablet_service.cc:1467] Processing CreateTablet for tablet addc32413df14a3082658f77f79271d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fef1945ec77d4e5787d64886b246ab52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:21:13.912290 12171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addc32413df14a3082658f77f79271d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:13.912292 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addc32413df14a3082658f77f79271d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:13.912556 12311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet addc32413df14a3082658f77f79271d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:13.919485 12401 tablet_bootstrap.cc:492] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: Bootstrap starting.
I20260620 02:21:13.919668 12402 tablet_bootstrap.cc:492] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Bootstrap starting.
I20260620 02:21:13.920104 12400 tablet_bootstrap.cc:492] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: Bootstrap starting.
I20260620 02:21:13.921417 12402 tablet_bootstrap.cc:654] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:13.921417 12401 tablet_bootstrap.cc:654] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:13.921519 12400 tablet_bootstrap.cc:654] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:13.924495 12401 tablet_bootstrap.cc:492] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: No bootstrap required, opened a new log
I20260620 02:21:13.924539 12402 tablet_bootstrap.cc:492] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: No bootstrap required, opened a new log
I20260620 02:21:13.924690 12400 tablet_bootstrap.cc:492] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: No bootstrap required, opened a new log
I20260620 02:21:13.924682 12402 ts_tablet_manager.cc:1397] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 02:21:13.924683 12401 ts_tablet_manager.cc:1397] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:21:13.924839 12400 ts_tablet_manager.cc:1397] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 02:21:13.925475 12401 raft_consensus.cc:348] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.925556 12402 raft_consensus.cc:348] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.925596 12400 raft_consensus.cc:348] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.925838 12402 raft_consensus.cc:374] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:13.925848 12400 raft_consensus.cc:374] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:13.925845 12401 raft_consensus.cc:374] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:13.925944 12400 raft_consensus.cc:729] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075c5a223b0845479bb74bf8b4e2d570, State: Initialized, Role: FOLLOWER
I20260620 02:21:13.925946 12401 raft_consensus.cc:729] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17065c9ddcf44de90aba68d08d779cb, State: Initialized, Role: FOLLOWER
I20260620 02:21:13.925944 12402 raft_consensus.cc:729] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d35f6d913cb4b3ea0f5b66b9d2902ac, State: Initialized, Role: FOLLOWER
I20260620 02:21:13.926175 12400 consensus_queue.cc:260] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.926184 12402 consensus_queue.cc:260] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.926785 12402 ts_tablet_manager.cc:1428] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:21:13.926826 12400 ts_tablet_manager.cc:1428] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:21:13.926666 12407 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:13.926994 12407 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:13.927363 12406 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:13.927768 12406 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:13.926250 12401 consensus_queue.cc:260] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.928256 12408 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:13.928629 12408 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:13.929154 12401 ts_tablet_manager.cc:1428] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:21:13.945669 12406 raft_consensus.cc:482] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:13.945917 12406 raft_consensus.cc:504] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.947714 12406 leader_election.cc:288] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167), d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913)
I20260620 02:21:13.952567 12408 raft_consensus.cc:482] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:13.952777 12408 raft_consensus.cc:504] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.954684 12408 leader_election.cc:288] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505), 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167)
I20260620 02:21:13.956836 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.948163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:60473 (local address 127.9.231.66:36167)
0620 02:21:13.948414 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.948424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.949165 (+ 741us) server_negotiation.cc:407] Connection header received
0620 02:21:13.949453 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.949459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.949533 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.949600 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.950290 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.951133 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.954126 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.955476 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.955639 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.955722 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.956350 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.956512 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:21:13.957984 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.948200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:36913 (local address 127.9.231.65:60703)
0620 02:21:13.949201 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.949220 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.949327 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.949690 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.949694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.949718 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.950112 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.950121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.951312 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.951316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.953951 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.953964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.956696 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.956705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.956965 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.957018 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.957128 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.957349 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":78,"thread_start_us":671,"threads_started":1}
I20260620 02:21:13.958329 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.948101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:36167 (local address 127.9.231.65:60473)
0620 02:21:13.949054 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.949071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.949240 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.949653 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.949658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.949681 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.950109 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.950118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.951311 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.951317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.953947 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.953964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.955885 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.955895 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.956142 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.956178 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.957520 (+ 1342us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.957724 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":57,"thread_start_us":512,"threads_started":1}
I20260620 02:21:13.959147 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.948233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:60703 (local address 127.9.231.67:36913)
0620 02:21:13.948415 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.948421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.949309 (+ 888us) server_negotiation.cc:407] Connection header received
0620 02:21:13.949467 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.949471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.949544 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.949655 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.950327 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.951166 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.954980 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.956385 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.956546 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.956611 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.958373 (+ 1762us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.958755 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:21:13.959517 12196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addc32413df14a3082658f77f79271d1" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570" is_pre_election: true
I20260620 02:21:13.959690 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addc32413df14a3082658f77f79271d1" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17065c9ddcf44de90aba68d08d779cb" is_pre_election: true
I20260620 02:21:13.959970 12336 raft_consensus.cc:2454] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 0.
I20260620 02:21:13.960011 12196 raft_consensus.cc:2454] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 0.
I20260620 02:21:13.960523 11982 leader_election.cc:302] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d35f6d913cb4b3ea0f5b66b9d2902ac, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:13.960987 12406 raft_consensus.cc:2781] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:13.961117 12406 raft_consensus.cc:482] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:13.961211 12406 raft_consensus.cc:3037] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:13.965353 12406 raft_consensus.cc:504] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.966354 12406 leader_election.cc:288] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 election: Requested vote from peers 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167), d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913)
I20260620 02:21:13.966956 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addc32413df14a3082658f77f79271d1" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17065c9ddcf44de90aba68d08d779cb"
I20260620 02:21:13.967160 12196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addc32413df14a3082658f77f79271d1" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570"
I20260620 02:21:13.967275 12336 raft_consensus.cc:3037] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:13.967389 12196 raft_consensus.cc:3037] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:13.967779 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.956861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:48923 (local address 127.9.231.66:36167)
0620 02:21:13.957079 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.957086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.957106 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:21:13.957186 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.957191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.957266 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.957358 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.960127 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.961268 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.964794 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.966502 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.966638 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.966709 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.967275 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.967439 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"threads_started":1}
I20260620 02:21:13.968417 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.954987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:36167 (local address 127.9.231.67:48923)
0620 02:21:13.956033 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.956058 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.956206 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.959449 (+ 3243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.959454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.959487 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.959871 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.959887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.961575 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.961584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.964536 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.964560 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.966866 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.966873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.967083 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.967119 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.967911 (+ 792us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.968074 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"threads_started":1}
I20260620 02:21:13.970542 12336 raft_consensus.cc:2454] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 1.
I20260620 02:21:13.970711 12196 raft_consensus.cc:2454] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 1.
I20260620 02:21:13.971035 11982 leader_election.cc:302] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d35f6d913cb4b3ea0f5b66b9d2902ac, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:13.971454 12406 raft_consensus.cc:2781] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:13.972028 12406 raft_consensus.cc:686] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 LEADER]: Becoming Leader. State: Replica: 7d35f6d913cb4b3ea0f5b66b9d2902ac, State: Running, Role: LEADER
I20260620 02:21:13.972759 12406 consensus_queue.cc:237] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:13.975033 12196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addc32413df14a3082658f77f79271d1" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570" is_pre_election: true
I20260620 02:21:13.975410 12196 raft_consensus.cc:2379] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d17065c9ddcf44de90aba68d08d779cb in current term 1: Already voted for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in this term.
I20260620 02:21:13.975673 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.954987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:39505 (local address 127.9.231.67:35967)
0620 02:21:13.968632 (+ 13645us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.968653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.968839 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.969373 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.969377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.969403 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:13.969563 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.969574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.970971 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.970977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.972744 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.972757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.974752 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.974758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.974956 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.974989 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:21:13.975128 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 02:21:13.975265 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13518,"mutex_wait_us":171,"thread_start_us":520,"threads_started":1}
I20260620 02:21:13.975761 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.955236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:35967 (local address 127.9.231.65:39505)
0620 02:21:13.957234 (+ 1998us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.957240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.968730 (+ 11490us) server_negotiation.cc:407] Connection header received
0620 02:21:13.968994 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.969000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.969077 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.969137 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:13.969746 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.970809 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.972921 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.974554 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.974752 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:13.974837 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:13.975177 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 02:21:13.975357 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886}
I20260620 02:21:13.976583 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "addc32413df14a3082658f77f79271d1" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" is_pre_election: true
I20260620 02:21:13.977409 12261 leader_election.cc:302] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d17065c9ddcf44de90aba68d08d779cb; no voters: 075c5a223b0845479bb74bf8b4e2d570, 7d35f6d913cb4b3ea0f5b66b9d2902ac
I20260620 02:21:13.977876 12408 raft_consensus.cc:2726] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 02:21:13.978627 12427 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: TxnStatusTablet state changed. Reason: New leader 7d35f6d913cb4b3ea0f5b66b9d2902ac. Latest consensus state: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:13.978981 12427 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:21:13.979720 12429 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:21:13.979529 11894 catalog_manager.cc:5495] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65). New cstate: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } health_report { overall_health: UNKNOWN } } }
I20260620 02:21:13.980027 12429 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:21:14.002745 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.995934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:39505 (local address 127.0.0.1:53520)
0620 02:21:13.996181 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:13.996203 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:13.996334 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:13.996907 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:13.996912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:13.996925 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:13.997088 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.997096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.998633 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.998637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:13.999766 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:13.999781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.000872 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.000877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.001028 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.001091 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.002216 (+ 1125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.002246 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:21:14.002327 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 02:21:14.002476 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 02:21:14.002835 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:13.995987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.9.231.65:39505)
0620 02:21:13.996182 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:13.996189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:13.996286 (+ 97us) server_negotiation.cc:407] Connection header received
0620 02:21:13.996555 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:13.996587 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:13.996716 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:13.996886 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:13.997220 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:13.998454 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:13.999928 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.000681 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.000828 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.001280 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.002099 (+ 819us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.002269 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:14.002383 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 02:21:14.002491 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:21:14.011199 12196 raft_consensus.cc:1261] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Refusing update from remote peer 7d35f6d913cb4b3ea0f5b66b9d2902ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:14.011231 12336 raft_consensus.cc:1261] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Refusing update from remote peer 7d35f6d913cb4b3ea0f5b66b9d2902ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:14.012179 12427 consensus_queue.cc:1035] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:14.012540 12406 consensus_queue.cc:1035] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:14.020694 12408 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: TxnStatusTablet state changed. Reason: New leader 7d35f6d913cb4b3ea0f5b66b9d2902ac. Latest consensus state: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:14.021018 12408 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:14.023728 12432 mvcc.cc:205] Tried to move back new op lower bound from 7298752815139364864 to 7298752814998487040. Current Snapshot: MvccSnapshot[applied={T|T < 7298752815139364864}]
I20260620 02:21:14.025220 12406 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:14.025641 12406 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:21:14.060863 12407 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: TxnStatusTablet state changed. Reason: New leader 7d35f6d913cb4b3ea0f5b66b9d2902ac. Latest consensus state: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:14.061372 12407 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:14.065773 12408 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:14.066082 12408 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P d17065c9ddcf44de90aba68d08d779cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:14.066329 12427 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:14.066617 12427 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:21:14.072499 12407 tablet_replica.cc:439] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } } }
I20260620 02:21:14.072821 12407 tablet_replica.cc:442] T addc32413df14a3082658f77f79271d1 P 075c5a223b0845479bb74bf8b4e2d570: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:21:14.075613 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.048296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.9.231.66:36167)
0620 02:21:14.048592 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:14.048601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:14.057965 (+ 9364us) server_negotiation.cc:407] Connection header received
0620 02:21:14.058451 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:14.058459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:14.058603 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:14.058790 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:14.061572 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.062915 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.071108 (+ 8193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.071992 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.072150 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.073117 (+ 967us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.074167 (+ 1050us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.074315 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:14.075090 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 02:21:14.075191 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:21:14.094069 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.087204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:39505 (local address 127.9.231.66:59373)
0620 02:21:14.087919 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:14.087936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:14.088059 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:14.088481 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:14.088485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:14.088496 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:14.088781 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.088788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.090334 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.090338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.091612 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.091627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.092762 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.092766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.092908 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.092973 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.093522 (+ 549us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.093535 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:21:14.093621 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:21:14.093753 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":378,"threads_started":1}
I20260620 02:21:14.094151 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.087407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:59373 (local address 127.9.231.65:39505)
0620 02:21:14.087626 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:14.087633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:14.088033 (+ 400us) server_negotiation.cc:407] Connection header received
0620 02:21:14.088194 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:14.088200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:14.088322 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:14.088454 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:14.088932 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.090116 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.091773 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.092569 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.092711 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.093184 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.093338 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.093447 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:14.093675 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 02:21:14.093778 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:21:14.105146 12056 raft_consensus.cc:1261] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Refusing update from remote peer 075c5a223b0845479bb74bf8b4e2d570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:14.106088 12336 raft_consensus.cc:1261] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Refusing update from remote peer 075c5a223b0845479bb74bf8b4e2d570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:14.105978 12411 consensus_queue.cc:1035] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:14.106709 12407 consensus_queue.cc:1035] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:14.112066 12434 mvcc.cc:205] Tried to move back new op lower bound from 7298752815527038976 to 7298752814311399424. Current Snapshot: MvccSnapshot[applied={T|T < 7298752815527038976}]
I20260620 02:21:14.120648 12433 mvcc.cc:205] Tried to move back new op lower bound from 7298752815527038976 to 7298752814311399424. Current Snapshot: MvccSnapshot[applied={T|T < 7298752815527038976}]
I20260620 02:21:14.147868 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.140240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:47865 (local address 127.9.231.66:36167)
0620 02:21:14.140428 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:14.140434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:14.141438 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 02:21:14.141638 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:14.141643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:14.141733 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:14.141847 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:14.142512 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.143785 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.145841 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.146433 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.146575 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.147142 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.147247 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.147352 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:14.147481 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 02:21:14.147559 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:21:14.148015 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.140229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:36167 (local address 127.9.231.65:47865)
0620 02:21:14.141321 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:14.141345 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:14.141490 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:14.141887 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:14.141892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:14.141906 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:14.142326 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.142341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.143929 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.143933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.145652 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.145665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.146615 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.146623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.146824 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.146910 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.147354 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.147372 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 02:21:14.147456 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 02:21:14.147602 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":544,"threads_started":1}
I20260620 02:21:14.188158 11893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:14.188452 11893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:14.192117 11893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Sending DeleteTablet for 3 replicas of tablet 847e6c74f4474dc69b63eb17e7513523
I20260620 02:21:14.193521 12311 tablet_service.cc:1514] Processing DeleteTablet for tablet 847e6c74f4474dc69b63eb17e7513523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:14 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 02:21:14.193946 12032 tablet_service.cc:1514] Processing DeleteTablet for tablet 847e6c74f4474dc69b63eb17e7513523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:14 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:53512
I20260620 02:21:14.194380 12171 tablet_service.cc:1514] Processing DeleteTablet for tablet 847e6c74f4474dc69b63eb17e7513523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:14 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:59498
I20260620 02:21:14.202828 11893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:14.203809 11893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:14.205107 12454 tablet_replica.cc:330] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: stopping tablet replica
I20260620 02:21:14.209506 12454 raft_consensus.cc:2229] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:14.210016 12454 raft_consensus.cc:2258] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:14.213737 12170 tablet_service.cc:1467] Processing CreateTablet for tablet 0427f988bec74710a2bc7ef0b1186f65 (DEFAULT_TABLE table=test-table [id=8ca0d5dbf1c042a7a29d3fe799573776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:14.214511 12170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0427f988bec74710a2bc7ef0b1186f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:14.216348 12310 tablet_service.cc:1467] Processing CreateTablet for tablet 0427f988bec74710a2bc7ef0b1186f65 (DEFAULT_TABLE table=test-table [id=8ca0d5dbf1c042a7a29d3fe799573776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:14.216691 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 0427f988bec74710a2bc7ef0b1186f65 (DEFAULT_TABLE table=test-table [id=8ca0d5dbf1c042a7a29d3fe799573776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:14.216905 12310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0427f988bec74710a2bc7ef0b1186f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:14.217319 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0427f988bec74710a2bc7ef0b1186f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:14.221282 12400 tablet_bootstrap.cc:492] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: Bootstrap starting.
I20260620 02:21:14.222962 12400 tablet_bootstrap.cc:654] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:14.223563 12401 tablet_bootstrap.cc:492] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: Bootstrap starting.
I20260620 02:21:14.224700 12454 ts_tablet_manager.cc:1905] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:14.225682 12401 tablet_bootstrap.cc:654] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:14.230628 12455 tablet_replica.cc:330] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: stopping tablet replica
I20260620 02:21:14.231197 12455 raft_consensus.cc:2229] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:14.231592 12402 tablet_bootstrap.cc:492] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Bootstrap starting.
I20260620 02:21:14.231947 12455 raft_consensus.cc:2258] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:14.232909 12453 tablet_replica.cc:330] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: stopping tablet replica
I20260620 02:21:14.233374 12453 raft_consensus.cc:2229] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:14.233556 12402 tablet_bootstrap.cc:654] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:14.233865 12453 raft_consensus.cc:2258] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:14.238989 12453 ts_tablet_manager.cc:1905] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:14.239620 12400 tablet_bootstrap.cc:492] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: No bootstrap required, opened a new log
I20260620 02:21:14.239816 12400 ts_tablet_manager.cc:1397] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20260620 02:21:14.240705 12400 raft_consensus.cc:348] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.240928 12401 tablet_bootstrap.cc:492] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: No bootstrap required, opened a new log
I20260620 02:21:14.240970 12400 raft_consensus.cc:374] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:14.241056 12400 raft_consensus.cc:729] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075c5a223b0845479bb74bf8b4e2d570, State: Initialized, Role: FOLLOWER
I20260620 02:21:14.241098 12401 ts_tablet_manager.cc:1397] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20260620 02:21:14.241367 12400 consensus_queue.cc:260] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.241874 12401 raft_consensus.cc:348] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.242112 12401 raft_consensus.cc:374] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:14.242198 12401 raft_consensus.cc:729] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17065c9ddcf44de90aba68d08d779cb, State: Initialized, Role: FOLLOWER
I20260620 02:21:14.242470 12401 consensus_queue.cc:260] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.244575 12400 ts_tablet_manager.cc:1428] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 02:21:14.245599 12401 ts_tablet_manager.cc:1428] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 02:21:14.246940 12402 tablet_bootstrap.cc:492] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: No bootstrap required, opened a new log
I20260620 02:21:14.247203 12402 ts_tablet_manager.cc:1397] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260620 02:21:14.248039 12402 raft_consensus.cc:348] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.248381 12402 raft_consensus.cc:374] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:14.248543 12402 raft_consensus.cc:729] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d35f6d913cb4b3ea0f5b66b9d2902ac, State: Initialized, Role: FOLLOWER
I20260620 02:21:14.248893 12402 consensus_queue.cc:260] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.251856 12402 ts_tablet_manager.cc:1428] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:21:14.257361 12454 ts_tablet_manager.cc:1918] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:14.258100 12455 ts_tablet_manager.cc:1905] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:14.258806 12454 log.cc:1196] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/847e6c74f4474dc69b63eb17e7513523
I20260620 02:21:14.259573 12454 ts_tablet_manager.cc:1939] T 847e6c74f4474dc69b63eb17e7513523 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting consensus metadata
I20260620 02:21:14.261605 11879 catalog_manager.cc:4841] TS 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505): tablet 847e6c74f4474dc69b63eb17e7513523 (table test-table [id=7d460e50aa4b4fa3b252942fe35cd432]) successfully deleted
I20260620 02:21:14.270726 12453 ts_tablet_manager.cc:1918] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:14.271046 12453 log.cc:1196] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/847e6c74f4474dc69b63eb17e7513523
I20260620 02:21:14.271718 12453 ts_tablet_manager.cc:1939] T 847e6c74f4474dc69b63eb17e7513523 P d17065c9ddcf44de90aba68d08d779cb: Deleting consensus metadata
I20260620 02:21:14.273347 12455 ts_tablet_manager.cc:1918] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:14.273867 11879 catalog_manager.cc:4841] TS d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913): tablet 847e6c74f4474dc69b63eb17e7513523 (table test-table [id=7d460e50aa4b4fa3b252942fe35cd432]) successfully deleted
I20260620 02:21:14.275178 12455 log.cc:1196] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/847e6c74f4474dc69b63eb17e7513523
I20260620 02:21:14.275802 12455 ts_tablet_manager.cc:1939] T 847e6c74f4474dc69b63eb17e7513523 P 075c5a223b0845479bb74bf8b4e2d570: Deleting consensus metadata
I20260620 02:21:14.277591 11881 catalog_manager.cc:4841] TS 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167): tablet 847e6c74f4474dc69b63eb17e7513523 (table test-table [id=7d460e50aa4b4fa3b252942fe35cd432]) successfully deleted
I20260620 02:21:14.469617 12408 raft_consensus.cc:482] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:14.469846 12408 raft_consensus.cc:504] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.470688 12408 leader_election.cc:288] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505), 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167)
I20260620 02:21:14.471467 12196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0427f988bec74710a2bc7ef0b1186f65" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570" is_pre_election: true
I20260620 02:21:14.471468 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0427f988bec74710a2bc7ef0b1186f65" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" is_pre_election: true
I20260620 02:21:14.471751 12196 raft_consensus.cc:2454] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 0.
I20260620 02:21:14.471755 12056 raft_consensus.cc:2454] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 0.
I20260620 02:21:14.472292 12263 leader_election.cc:302] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:14.472652 12408 raft_consensus.cc:2781] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:14.472779 12408 raft_consensus.cc:482] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:14.472899 12408 raft_consensus.cc:3037] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:14.476197 12408 raft_consensus.cc:504] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.476955 12408 leader_election.cc:288] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 election: Requested vote from peers 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505), 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167)
I20260620 02:21:14.477516 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0427f988bec74710a2bc7ef0b1186f65" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac"
I20260620 02:21:14.477690 12196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0427f988bec74710a2bc7ef0b1186f65" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570"
I20260620 02:21:14.477802 12056 raft_consensus.cc:3037] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:14.477916 12196 raft_consensus.cc:3037] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:14.481107 12056 raft_consensus.cc:2454] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 1.
I20260620 02:21:14.481269 12196 raft_consensus.cc:2454] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 1.
I20260620 02:21:14.481801 12261 leader_election.cc:302] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, 7d35f6d913cb4b3ea0f5b66b9d2902ac, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:14.482424 12408 raft_consensus.cc:2781] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:14.482856 12408 raft_consensus.cc:686] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 1 LEADER]: Becoming Leader. State: Replica: d17065c9ddcf44de90aba68d08d779cb, State: Running, Role: LEADER
I20260620 02:21:14.483314 12408 consensus_queue.cc:237] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:14.487265 11893 catalog_manager.cc:5495] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb reported cstate change: term changed from 0 to 1, leader changed from <none> to d17065c9ddcf44de90aba68d08d779cb (127.9.231.67). New cstate: current_term: 1 leader_uuid: "d17065c9ddcf44de90aba68d08d779cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } health_report { overall_health: HEALTHY } } }
I20260620 02:21:14.540629 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.533133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.9.231.67:36913)
0620 02:21:14.533747 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:14.533753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:14.533773 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:21:14.533898 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:14.533903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:14.534034 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:14.534206 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:14.534647 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.535723 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.537758 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.538416 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.538653 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.539198 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.539924 (+ 726us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.540085 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:14.540206 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 02:21:14.540295 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":323,"threads_started":1}
I20260620 02:21:14.558499 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.552205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:39505 (local address 127.9.231.67:46057)
0620 02:21:14.552887 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:14.552902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:14.553086 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:14.553512 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:14.553516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:14.553529 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:14.553807 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.553815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.554943 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.554948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.556294 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.556309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.557168 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.557174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.557366 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.557452 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.557873 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.557886 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:21:14.557971 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 02:21:14.558123 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":367,"threads_started":1}
I20260620 02:21:14.558422 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.552262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:46057 (local address 127.9.231.65:39505)
0620 02:21:14.552453 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:14.552461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:14.553014 (+ 553us) server_negotiation.cc:407] Connection header received
0620 02:21:14.553267 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:14.553275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:14.553377 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:14.553487 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:14.553955 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.554804 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.556446 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.557010 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.557153 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.557654 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.557775 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.557872 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:14.558008 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 02:21:14.558090 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:21:14.572310 12196 raft_consensus.cc:1261] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Refusing update from remote peer d17065c9ddcf44de90aba68d08d779cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:14.573040 12408 consensus_queue.cc:1035] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:14.574501 12056 raft_consensus.cc:1261] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Refusing update from remote peer d17065c9ddcf44de90aba68d08d779cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:14.575150 12459 consensus_queue.cc:1035] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:14.606191 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.599319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:36913 (local address 127.9.231.65:34251)
0620 02:21:14.599723 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:14.599742 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:14.599854 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:14.600684 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:14.600688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:14.600700 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:14.600927 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.600941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.602316 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.602323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.604031 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:14.604043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.604890 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.604897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:14.605080 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.605150 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.605629 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.605639 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:21:14.605712 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 02:21:14.605864 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260620 02:21:14.606890 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:14.600076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:34251 (local address 127.9.231.67:36913)
0620 02:21:14.600226 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:14.600233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:14.600254 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:21:14.600306 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:14.600309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:14.600414 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:14.600499 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:14.601083 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.602168 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.604181 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:14.604718 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:14.604861 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:14.605329 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:14.605490 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:14.605576 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:14.606514 (+ 938us) server_negotiation.cc:299] Negotiation successful
0620 02:21:14.606601 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 02:21:14.608338 12433 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:21:16.232745 11893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:16.232980 11893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:16.237630 11893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Sending DeleteTablet for 3 replicas of tablet 0427f988bec74710a2bc7ef0b1186f65
I20260620 02:21:16.238631 12171 tablet_service.cc:1514] Processing DeleteTablet for tablet 0427f988bec74710a2bc7ef0b1186f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:16 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:59498
I20260620 02:21:16.238703 12032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0427f988bec74710a2bc7ef0b1186f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:16 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:53512
I20260620 02:21:16.238763 12311 tablet_service.cc:1514] Processing DeleteTablet for tablet 0427f988bec74710a2bc7ef0b1186f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:16 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 02:21:16.254119 12492 tablet_replica.cc:330] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: stopping tablet replica
I20260620 02:21:16.254343 12492 raft_consensus.cc:2229] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:16.254599 12492 raft_consensus.cc:2258] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:16.254622 12493 tablet_replica.cc:330] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: stopping tablet replica
I20260620 02:21:16.255229 12491 tablet_replica.cc:330] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: stopping tablet replica
I20260620 02:21:16.255342 12493 raft_consensus.cc:2229] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:16.255420 12491 raft_consensus.cc:2229] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:16.255659 12491 raft_consensus.cc:2258] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:16.255941 12493 raft_consensus.cc:2258] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:16.256562 12492 ts_tablet_manager.cc:1905] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:16.257247 12491 ts_tablet_manager.cc:1905] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:16.263444 12493 ts_tablet_manager.cc:1905] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:16.265298 12491 ts_tablet_manager.cc:1918] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:21:16.265585 12491 log.cc:1196] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/0427f988bec74710a2bc7ef0b1186f65
I20260620 02:21:16.266315 12491 ts_tablet_manager.cc:1939] T 0427f988bec74710a2bc7ef0b1186f65 P 075c5a223b0845479bb74bf8b4e2d570: Deleting consensus metadata
I20260620 02:21:16.267864 12492 ts_tablet_manager.cc:1918] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:21:16.268009 12492 log.cc:1196] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/0427f988bec74710a2bc7ef0b1186f65
I20260620 02:21:16.267998 11881 catalog_manager.cc:4841] TS 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167): tablet 0427f988bec74710a2bc7ef0b1186f65 (table test-table [id=8ca0d5dbf1c042a7a29d3fe799573776]) successfully deleted
I20260620 02:21:16.268440 12492 ts_tablet_manager.cc:1939] T 0427f988bec74710a2bc7ef0b1186f65 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting consensus metadata
I20260620 02:21:16.269692 11879 catalog_manager.cc:4841] TS 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505): tablet 0427f988bec74710a2bc7ef0b1186f65 (table test-table [id=8ca0d5dbf1c042a7a29d3fe799573776]) successfully deleted
I20260620 02:21:16.270816 12493 ts_tablet_manager.cc:1918] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:21:16.270946 12493 log.cc:1196] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/0427f988bec74710a2bc7ef0b1186f65
I20260620 02:21:16.271431 12493 ts_tablet_manager.cc:1939] T 0427f988bec74710a2bc7ef0b1186f65 P d17065c9ddcf44de90aba68d08d779cb: Deleting consensus metadata
I20260620 02:21:16.272552 11879 catalog_manager.cc:4841] TS d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913): tablet 0427f988bec74710a2bc7ef0b1186f65 (table test-table [id=8ca0d5dbf1c042a7a29d3fe799573776]) successfully deleted
Jun 20 02:21:16 dist-test-slave-9vg1 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:21:16 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:16.306397 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:16.290223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.9.231.126:44557)
0620 02:21:16.290839 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:16.290843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:16.290868 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:21:16.291022 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:16.291026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:16.291155 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:16.291300 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:16.292891 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:16.293710 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:16.295262 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:16.295700 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:16.295806 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:16.300247 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:16.300253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:16.300257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:16.300298 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:16.303556 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:16.304283 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:16.304288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:16.304291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:16.304513 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:16.304921 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:16.304926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:16.304928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:16.305290 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:16.305495 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:16.305835 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 02:21:16.306037 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":338,"threads_started":1}
I20260620 02:21:16.310735 11893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:16.311419 11893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:16.319139 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 1c05a11329b84726b16e0c26a95418a6 (DEFAULT_TABLE table=test-table [id=749fdf844f174e929231b6be046b82e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:16.319409 12171 tablet_service.cc:1467] Processing CreateTablet for tablet 1c05a11329b84726b16e0c26a95418a6 (DEFAULT_TABLE table=test-table [id=749fdf844f174e929231b6be046b82e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:16.319635 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c05a11329b84726b16e0c26a95418a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:16.319572 12311 tablet_service.cc:1467] Processing CreateTablet for tablet 1c05a11329b84726b16e0c26a95418a6 (DEFAULT_TABLE table=test-table [id=749fdf844f174e929231b6be046b82e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:16.319810 12171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c05a11329b84726b16e0c26a95418a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:16.320009 12311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c05a11329b84726b16e0c26a95418a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:16.327710 12503 tablet_bootstrap.cc:492] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Bootstrap starting.
I20260620 02:21:16.328097 12504 tablet_bootstrap.cc:492] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: Bootstrap starting.
I20260620 02:21:16.327910 12502 tablet_bootstrap.cc:492] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: Bootstrap starting.
I20260620 02:21:16.330188 12503 tablet_bootstrap.cc:654] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:16.330559 12504 tablet_bootstrap.cc:654] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:16.331584 12502 tablet_bootstrap.cc:654] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:16.333097 12503 tablet_bootstrap.cc:492] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: No bootstrap required, opened a new log
I20260620 02:21:16.333388 12503 ts_tablet_manager.cc:1397] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 02:21:16.335139 12503 raft_consensus.cc:348] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.335564 12503 raft_consensus.cc:374] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:16.335824 12503 raft_consensus.cc:729] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d35f6d913cb4b3ea0f5b66b9d2902ac, State: Initialized, Role: FOLLOWER
I20260620 02:21:16.336398 12502 tablet_bootstrap.cc:492] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: No bootstrap required, opened a new log
I20260620 02:21:16.336280 12503 consensus_queue.cc:260] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.336572 12502 ts_tablet_manager.cc:1397] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 02:21:16.337054 12503 ts_tablet_manager.cc:1428] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:21:16.337608 12502 raft_consensus.cc:348] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.337893 12502 raft_consensus.cc:374] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:16.338012 12502 raft_consensus.cc:729] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075c5a223b0845479bb74bf8b4e2d570, State: Initialized, Role: FOLLOWER
I20260620 02:21:16.338299 12502 consensus_queue.cc:260] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.339968 12504 tablet_bootstrap.cc:492] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: No bootstrap required, opened a new log
I20260620 02:21:16.340164 12504 ts_tablet_manager.cc:1397] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260620 02:21:16.341137 12504 raft_consensus.cc:348] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.341441 12504 raft_consensus.cc:374] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:16.341567 12504 raft_consensus.cc:729] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17065c9ddcf44de90aba68d08d779cb, State: Initialized, Role: FOLLOWER
I20260620 02:21:16.341842 12504 consensus_queue.cc:260] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.342432 12504 ts_tablet_manager.cc:1428] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:21:16.343041 12502 ts_tablet_manager.cc:1428] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260620 02:21:16.729064 12486 raft_consensus.cc:482] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:16.729264 12486 raft_consensus.cc:504] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.729907 12486 leader_election.cc:288] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505), 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167)
I20260620 02:21:16.730611 12196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c05a11329b84726b16e0c26a95418a6" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570" is_pre_election: true
I20260620 02:21:16.730648 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c05a11329b84726b16e0c26a95418a6" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" is_pre_election: true
I20260620 02:21:16.730868 12196 raft_consensus.cc:2454] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 0.
I20260620 02:21:16.730926 12056 raft_consensus.cc:2454] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 0.
I20260620 02:21:16.731531 12263 leader_election.cc:302] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, 7d35f6d913cb4b3ea0f5b66b9d2902ac, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:16.731842 12486 raft_consensus.cc:2781] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:16.731966 12486 raft_consensus.cc:482] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:16.732081 12486 raft_consensus.cc:3037] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:16.734123 12486 raft_consensus.cc:504] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.734758 12486 leader_election.cc:288] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 election: Requested vote from peers 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505), 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167)
I20260620 02:21:16.735383 12056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c05a11329b84726b16e0c26a95418a6" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac"
I20260620 02:21:16.735397 12196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c05a11329b84726b16e0c26a95418a6" candidate_uuid: "d17065c9ddcf44de90aba68d08d779cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570"
I20260620 02:21:16.735620 12196 raft_consensus.cc:3037] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:16.735621 12056 raft_consensus.cc:3037] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:16.737649 12056 raft_consensus.cc:2454] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 1.
I20260620 02:21:16.737835 12196 raft_consensus.cc:2454] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d17065c9ddcf44de90aba68d08d779cb in term 1.
I20260620 02:21:16.738206 12261 leader_election.cc:302] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, 7d35f6d913cb4b3ea0f5b66b9d2902ac, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:16.738509 12486 raft_consensus.cc:2781] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:16.738886 12486 raft_consensus.cc:686] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 1 LEADER]: Becoming Leader. State: Replica: d17065c9ddcf44de90aba68d08d779cb, State: Running, Role: LEADER
I20260620 02:21:16.739221 12486 consensus_queue.cc:237] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.741789 11893 catalog_manager.cc:5495] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb reported cstate change: term changed from 0 to 1, leader changed from <none> to d17065c9ddcf44de90aba68d08d779cb (127.9.231.67). New cstate: current_term: 1 leader_uuid: "d17065c9ddcf44de90aba68d08d779cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } health_report { overall_health: HEALTHY } } }
I20260620 02:21:16.803987 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:16.797347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.9.231.67:36913)
0620 02:21:16.797874 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:16.797878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:16.797895 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:21:16.797953 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:16.797957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:16.798027 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:16.798118 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:16.798592 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:16.799672 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:16.801676 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:16.802162 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:16.802296 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:16.802884 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:16.803022 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:16.803216 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:16.803402 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 02:21:16.803483 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":293,"threads_started":1}
I20260620 02:21:16.814922 12196 raft_consensus.cc:1261] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Refusing update from remote peer d17065c9ddcf44de90aba68d08d779cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:16.815408 12056 raft_consensus.cc:1261] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Refusing update from remote peer d17065c9ddcf44de90aba68d08d779cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:16.815500 12486 consensus_queue.cc:1035] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:16.816006 12486 consensus_queue.cc:1035] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:16.873258 11894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:16.873526 11894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:16.876855 11894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Sending DeleteTablet for 3 replicas of tablet 1c05a11329b84726b16e0c26a95418a6
I20260620 02:21:16.877753 12171 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c05a11329b84726b16e0c26a95418a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:16 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:59498
I20260620 02:21:16.877861 12032 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c05a11329b84726b16e0c26a95418a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:16 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:53512
I20260620 02:21:16.877880 12311 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c05a11329b84726b16e0c26a95418a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:16 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 02:21:16.879312 11894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:16.880319 11894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:16.888065 12529 tablet_replica.cc:330] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: stopping tablet replica
I20260620 02:21:16.890512 12170 tablet_service.cc:1467] Processing CreateTablet for tablet 3da89dea6361480ba0838c55f88fd7f7 (DEFAULT_TABLE table=test-table [id=dd5e2fe0d7d7482fa9687024c24ae5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:16.890729 12310 tablet_service.cc:1467] Processing CreateTablet for tablet 3da89dea6361480ba0838c55f88fd7f7 (DEFAULT_TABLE table=test-table [id=dd5e2fe0d7d7482fa9687024c24ae5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:16.890973 12170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da89dea6361480ba0838c55f88fd7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:16.891168 12310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da89dea6361480ba0838c55f88fd7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:16.898434 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 3da89dea6361480ba0838c55f88fd7f7 (DEFAULT_TABLE table=test-table [id=dd5e2fe0d7d7482fa9687024c24ae5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:16.888507 12529 raft_consensus.cc:2229] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:16.902091 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da89dea6361480ba0838c55f88fd7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:16.902469 12529 raft_consensus.cc:2258] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:16.903663 12528 tablet_replica.cc:330] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: stopping tablet replica
I20260620 02:21:16.903967 12528 raft_consensus.cc:2229] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:16.904050 12529 ts_tablet_manager.cc:1905] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:16.904425 12528 raft_consensus.cc:2258] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:16.905057 12527 tablet_replica.cc:330] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: stopping tablet replica
I20260620 02:21:16.905314 12527 raft_consensus.cc:2229] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:16.905714 12527 raft_consensus.cc:2258] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:16.905766 12528 ts_tablet_manager.cc:1905] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:16.906283 12530 tablet_bootstrap.cc:492] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: Bootstrap starting.
I20260620 02:21:16.909435 12530 tablet_bootstrap.cc:654] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:16.910892 12527 ts_tablet_manager.cc:1905] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:16.915336 12528 ts_tablet_manager.cc:1918] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:16.916285 12528 log.cc:1196] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/1c05a11329b84726b16e0c26a95418a6
I20260620 02:21:16.916884 12528 ts_tablet_manager.cc:1939] T 1c05a11329b84726b16e0c26a95418a6 P d17065c9ddcf44de90aba68d08d779cb: Deleting consensus metadata
I20260620 02:21:16.916949 12532 tablet_bootstrap.cc:492] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Bootstrap starting.
I20260620 02:21:16.917050 12530 tablet_bootstrap.cc:492] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: No bootstrap required, opened a new log
I20260620 02:21:16.917269 12530 ts_tablet_manager.cc:1397] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 02:21:16.918249 11879 catalog_manager.cc:4841] TS d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913): tablet 1c05a11329b84726b16e0c26a95418a6 (table test-table [id=749fdf844f174e929231b6be046b82e0]) successfully deleted
I20260620 02:21:16.918061 12530 raft_consensus.cc:348] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.918705 12530 raft_consensus.cc:374] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:16.918737 12532 tablet_bootstrap.cc:654] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:16.918756 12527 ts_tablet_manager.cc:1918] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:16.918805 12530 raft_consensus.cc:729] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17065c9ddcf44de90aba68d08d779cb, State: Initialized, Role: FOLLOWER
I20260620 02:21:16.918900 12527 log.cc:1196] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/1c05a11329b84726b16e0c26a95418a6
I20260620 02:21:16.919051 12530 consensus_queue.cc:260] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.919445 12527 ts_tablet_manager.cc:1939] T 1c05a11329b84726b16e0c26a95418a6 P 075c5a223b0845479bb74bf8b4e2d570: Deleting consensus metadata
I20260620 02:21:16.920413 12531 tablet_bootstrap.cc:492] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: Bootstrap starting.
I20260620 02:21:16.920661 12530 ts_tablet_manager.cc:1428] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:21:16.920647 11881 catalog_manager.cc:4841] TS 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167): tablet 1c05a11329b84726b16e0c26a95418a6 (table test-table [id=749fdf844f174e929231b6be046b82e0]) successfully deleted
I20260620 02:21:16.922796 12529 ts_tablet_manager.cc:1918] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:21:16.923029 12529 log.cc:1196] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/1c05a11329b84726b16e0c26a95418a6
I20260620 02:21:16.923593 12529 ts_tablet_manager.cc:1939] T 1c05a11329b84726b16e0c26a95418a6 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting consensus metadata
I20260620 02:21:16.923647 12532 tablet_bootstrap.cc:492] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: No bootstrap required, opened a new log
I20260620 02:21:16.923911 12532 ts_tablet_manager.cc:1397] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 02:21:16.924227 12531 tablet_bootstrap.cc:654] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:16.924687 11879 catalog_manager.cc:4841] TS 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505): tablet 1c05a11329b84726b16e0c26a95418a6 (table test-table [id=749fdf844f174e929231b6be046b82e0]) successfully deleted
I20260620 02:21:16.924813 12532 raft_consensus.cc:348] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.925101 12532 raft_consensus.cc:374] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:16.925226 12532 raft_consensus.cc:729] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d35f6d913cb4b3ea0f5b66b9d2902ac, State: Initialized, Role: FOLLOWER
I20260620 02:21:16.925480 12532 consensus_queue.cc:260] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.926066 12532 ts_tablet_manager.cc:1428] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:21:16.926697 12531 tablet_bootstrap.cc:492] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: No bootstrap required, opened a new log
I20260620 02:21:16.926851 12531 ts_tablet_manager.cc:1397] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 02:21:16.927716 12531 raft_consensus.cc:348] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.928011 12531 raft_consensus.cc:374] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:16.928128 12531 raft_consensus.cc:729] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075c5a223b0845479bb74bf8b4e2d570, State: Initialized, Role: FOLLOWER
I20260620 02:21:16.928400 12531 consensus_queue.cc:260] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.929553 12531 ts_tablet_manager.cc:1428] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 02:21:16.974197 12521 raft_consensus.cc:482] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:16.974417 12521 raft_consensus.cc:504] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.975376 12521 leader_election.cc:288] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167), d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913)
I20260620 02:21:16.975956 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da89dea6361480ba0838c55f88fd7f7" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17065c9ddcf44de90aba68d08d779cb" is_pre_election: true
I20260620 02:21:16.975972 12195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da89dea6361480ba0838c55f88fd7f7" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570" is_pre_election: true
I20260620 02:21:16.976204 12336 raft_consensus.cc:2454] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 0.
I20260620 02:21:16.976226 12195 raft_consensus.cc:2454] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 0.
I20260620 02:21:16.976771 11982 leader_election.cc:302] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, 7d35f6d913cb4b3ea0f5b66b9d2902ac, d17065c9ddcf44de90aba68d08d779cb; no voters:
I20260620 02:21:16.977088 12521 raft_consensus.cc:2781] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:16.977211 12521 raft_consensus.cc:482] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:16.977350 12521 raft_consensus.cc:3037] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:16.979732 12521 raft_consensus.cc:504] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.980367 12521 leader_election.cc:288] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 election: Requested vote from peers 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167), d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913)
I20260620 02:21:16.980926 12195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da89dea6361480ba0838c55f88fd7f7" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075c5a223b0845479bb74bf8b4e2d570"
I20260620 02:21:16.981016 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da89dea6361480ba0838c55f88fd7f7" candidate_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d17065c9ddcf44de90aba68d08d779cb"
I20260620 02:21:16.981179 12195 raft_consensus.cc:3037] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:16.981256 12336 raft_consensus.cc:3037] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:16.984365 12195 raft_consensus.cc:2454] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 1.
I20260620 02:21:16.984356 12336 raft_consensus.cc:2454] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d35f6d913cb4b3ea0f5b66b9d2902ac in term 1.
I20260620 02:21:16.984848 11984 leader_election.cc:302] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075c5a223b0845479bb74bf8b4e2d570, 7d35f6d913cb4b3ea0f5b66b9d2902ac; no voters:
I20260620 02:21:16.985301 12521 raft_consensus.cc:2781] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:16.985534 12521 raft_consensus.cc:686] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 LEADER]: Becoming Leader. State: Replica: 7d35f6d913cb4b3ea0f5b66b9d2902ac, State: Running, Role: LEADER
I20260620 02:21:16.985837 12521 consensus_queue.cc:237] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } }
I20260620 02:21:16.988897 11893 catalog_manager.cc:5495] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65). New cstate: current_term: 1 leader_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d35f6d913cb4b3ea0f5b66b9d2902ac" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 39505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 } health_report { overall_health: UNKNOWN } } }
I20260620 02:21:17.018859 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:17.013036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.9.231.65:39505)
0620 02:21:17.013729 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:17.013733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:17.013753 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:21:17.013811 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:17.013815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:17.013880 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:17.013981 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:17.014529 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:17.015622 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:17.017083 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:17.017551 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:17.017665 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:17.018123 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:17.018234 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:17.018316 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:17.018498 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 02:21:17.018563 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":431,"threads_started":1}
I20260620 02:21:17.034091 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:17.028507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:47261 (local address 127.9.231.65:39505)
0620 02:21:17.028752 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:17.028757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:17.028982 (+ 225us) server_negotiation.cc:407] Connection header received
0620 02:21:17.029131 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:17.029135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:17.029201 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:17.029283 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:17.029777 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:17.030742 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:17.032290 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:17.032897 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:17.032997 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:17.033362 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:17.033459 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:17.033562 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:17.033741 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 02:21:17.033838 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 02:21:17.034260 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:17.028144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:39505 (local address 127.9.231.65:47261)
0620 02:21:17.028884 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:17.028901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:17.029025 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:17.029372 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:17.029376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:17.029386 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:21:17.029630 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:17.029638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:17.030908 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:17.030912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:17.032143 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:17.032156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:17.033059 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:17.033065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:17.033181 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:17.033217 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:21:17.033595 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:21:17.033604 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:21:17.033708 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:21:17.033824 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":447,"threads_started":1}
I20260620 02:21:17.042765 12336 raft_consensus.cc:1261] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Refusing update from remote peer 7d35f6d913cb4b3ea0f5b66b9d2902ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:17.042801 12195 raft_consensus.cc:1261] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Refusing update from remote peer 7d35f6d913cb4b3ea0f5b66b9d2902ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:17.043439 12537 consensus_queue.cc:1035] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "075c5a223b0845479bb74bf8b4e2d570" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:17.043766 12521 consensus_queue.cc:1035] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d17065c9ddcf44de90aba68d08d779cb" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:21:17.066574 12511 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:21:18.663367 11894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:18.663681 11894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:18.668071 11894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 267a55f972564abebaef8736f68013cd: Sending DeleteTablet for 3 replicas of tablet 3da89dea6361480ba0838c55f88fd7f7
I20260620 02:21:18.669106 12032 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da89dea6361480ba0838c55f88fd7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:18 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:53512
I20260620 02:21:18.669124 12170 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da89dea6361480ba0838c55f88fd7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:18 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:59498
I20260620 02:21:18.669271 12310 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da89dea6361480ba0838c55f88fd7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:18 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 02:21:18.670509 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11966
I20260620 02:21:18.681850 12566 tablet_replica.cc:330] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: stopping tablet replica
I20260620 02:21:18.682360 12566 raft_consensus.cc:2229] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:18.682778 12566 raft_consensus.cc:2258] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:18.683693 12565 tablet_replica.cc:330] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: stopping tablet replica
I20260620 02:21:18.684103 12565 raft_consensus.cc:2229] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:18.684221 12566 ts_tablet_manager.cc:1905] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:18.684715 12565 raft_consensus.cc:2258] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:18.685837 12565 ts_tablet_manager.cc:1905] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:18.687811 12564 tablet_replica.cc:330] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: stopping tablet replica
I20260620 02:21:18.688091 12564 raft_consensus.cc:2229] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:18.688735 12564 raft_consensus.cc:2258] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:18.690039 12564 ts_tablet_manager.cc:1905] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:18.692627 12566 ts_tablet_manager.cc:1918] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:21:18.693024 12565 ts_tablet_manager.cc:1918] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:21:18.693151 12565 log.cc:1196] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/3da89dea6361480ba0838c55f88fd7f7
I20260620 02:21:18.693169 12566 log.cc:1196] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/3da89dea6361480ba0838c55f88fd7f7
I20260620 02:21:18.693616 12565 ts_tablet_manager.cc:1939] T 3da89dea6361480ba0838c55f88fd7f7 P 075c5a223b0845479bb74bf8b4e2d570: Deleting consensus metadata
I20260620 02:21:18.693616 12566 ts_tablet_manager.cc:1939] T 3da89dea6361480ba0838c55f88fd7f7 P d17065c9ddcf44de90aba68d08d779cb: Deleting consensus metadata
I20260620 02:21:18.694809 11881 catalog_manager.cc:4841] TS 075c5a223b0845479bb74bf8b4e2d570 (127.9.231.66:36167): tablet 3da89dea6361480ba0838c55f88fd7f7 (table test-table [id=dd5e2fe0d7d7482fa9687024c24ae5d6]) successfully deleted
I20260620 02:21:18.694809 11879 catalog_manager.cc:4841] TS d17065c9ddcf44de90aba68d08d779cb (127.9.231.67:36913): tablet 3da89dea6361480ba0838c55f88fd7f7 (table test-table [id=dd5e2fe0d7d7482fa9687024c24ae5d6]) successfully deleted
I20260620 02:21:18.696393 12564 ts_tablet_manager.cc:1918] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:21:18.696573 12564 log.cc:1196] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/3da89dea6361480ba0838c55f88fd7f7
I20260620 02:21:18.697028 12564 ts_tablet_manager.cc:1939] T 3da89dea6361480ba0838c55f88fd7f7 P 7d35f6d913cb4b3ea0f5b66b9d2902ac: Deleting consensus metadata
I20260620 02:21:18.698043 11879 catalog_manager.cc:4841] TS 7d35f6d913cb4b3ea0f5b66b9d2902ac (127.9.231.65:39505): tablet 3da89dea6361480ba0838c55f88fd7f7 (table test-table [id=dd5e2fe0d7d7482fa9687024c24ae5d6]) successfully deleted
Jun 20 02:21:18 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:18.708930 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:18.672031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.9.231.65:39505)
0620 02:21:18.684698 (+ 12667us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:18.684704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:18.684730 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:21:18.684798 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:18.684806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:18.684907 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:18.685056 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:18.687715 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:18.688873 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:18.690859 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:18.691883 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:18.692011 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:18.702865 (+ 10854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:18.702883 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:18.702903 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:18.702949 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:18.705938 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:18.706669 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:18.706677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:18.706679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:18.706812 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:18.707286 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:18.707291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:18.707293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:18.707689 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:18.708015 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:18.708317 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 02:21:18.708559 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12508,"thread_start_us":12358,"threads_started":1}
W20260620 02:21:18.709707 12097 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42738
W20260620 02:21:18.710310 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:18.710510 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 11966
I20260620 02:21:18.737533 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12106
Jun 20 02:21:18 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:21:18.754455 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:18.738410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.9.231.66:36167)
0620 02:21:18.738974 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:18.738978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:18.738995 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:21:18.739050 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:18.739097 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:18.739173 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:18.739267 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:18.740530 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:18.741281 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:18.742485 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:18.742891 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:18.742984 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:18.748006 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:18.748024 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:18.748041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:18.748091 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:18.751297 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:18.752005 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:18.752014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:18.752016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:18.752159 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:18.752624 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:18.752628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:18.752631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:18.753091 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:18.753373 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:18.753790 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 02:21:18.754048 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":311,"threads_started":1}
W20260620 02:21:18.755278 12236 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49044
W20260620 02:21:18.756135 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12106: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:18.756340 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 12106
I20260620 02:21:18.779583 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12245
Jun 20 02:21:18 dist-test-slave-9vg1 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:21:18.796494 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:18.780507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.9.231.67:36913)
0620 02:21:18.781023 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:18.781028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:18.781045 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:21:18.781102 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:18.781105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:18.781162 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:18.781271 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:18.782569 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:18.783495 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:18.785024 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:18.785485 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:18.785586 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:18.790233 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:18.790250 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:18.790269 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:18.790308 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:18.793374 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:18.794138 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:18.794147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:18.794149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:18.794291 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:18.794760 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:18.794766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:18.794769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:18.795289 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:18.795610 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:18.795898 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 02:21:18.796147 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":264,"threads_started":1}
W20260620 02:21:18.797303 12376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35956
W20260620 02:21:18.798060 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12245: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:18.798273 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 12245
I20260620 02:21:18.822172 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 11863
I20260620 02:21:18.823643 11935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:18.947921 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 11863
2026-06-20T02:21:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9141 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:19 dist-test-slave-9vg1 krb5kdc[12574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:19 dist-test-slave-9vg1 krb5kdc[12574](info): set up 2 sockets
Jun 20 02:21:19 dist-test-slave-9vg1 krb5kdc[12574](info): commencing operation
krb5kdc: starting...
W20260620 02:21:20.163254 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.009s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 02:21:20.410815 10141 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.247s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 02:21:20.591913 10141 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.181s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 02:21:20.794292 10141 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.202s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/test-user.keytab.
W20260620 02:21:21.446043 10141 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.651s user 0.000s sys 0.005s
Jun 20 02:21:21 dist-test-slave-9vg1 krb5kdc[12574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
W20260620 02:21:21.678058 10141 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.231.126: real 0.176s user 0.000s sys 0.010s
I20260620 02:21:21.679178 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33673
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:22.026039 12587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:22.026436 12587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:22.026614 12587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:22.034152 12587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:22.034353 12587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:22.034475 12587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:22.034590 12587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:22.034678 12587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:22.045050 12587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:22.046532 12587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:22.048051 12587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:22.061439 12592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:22.061471 12593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:22.061455 12595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:22.061875 12587 server_base.cc:1034] running on GCE node
I20260620 02:21:22.062739 12587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:22.064420 12587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:22.065645 12587 hybrid_clock.cc:648] HybridClock initialized: now 1781922082065601 us; error 34 us; skew 500 ppm
I20260620 02:21:22.066080 12587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:22 dist-test-slave-9vg1 krb5kdc[12574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922082, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:22.069892 12587 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:22.071595 12587 webserver.cc:460] Webserver started at http://127.9.231.126:35741/ using document root <none> and password file <none>
I20260620 02:21:22.072305 12587 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:22.072407 12587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:22.072682 12587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:22.075901 12587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "750f85bbc46e4c9eba94f373592ffa3c"
format_stamp: "Formatted at 2026-06-20 02:21:22 on dist-test-slave-9vg1"
server_key: "6893e2897774a12fcf6dec5ed76b4636"
server_key_iv: "837ad098015d86d8c6ba7292e538df2e"
server_key_version: "encryptionkey@0"
I20260620 02:21:22.076767 12587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "750f85bbc46e4c9eba94f373592ffa3c"
format_stamp: "Formatted at 2026-06-20 02:21:22 on dist-test-slave-9vg1"
server_key: "6893e2897774a12fcf6dec5ed76b4636"
server_key_iv: "837ad098015d86d8c6ba7292e538df2e"
server_key_version: "encryptionkey@0"
I20260620 02:21:22.082664 12587 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:21:22.086710 12602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:22.088481 12587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:22.088701 12587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "750f85bbc46e4c9eba94f373592ffa3c"
format_stamp: "Formatted at 2026-06-20 02:21:22 on dist-test-slave-9vg1"
server_key: "6893e2897774a12fcf6dec5ed76b4636"
server_key_iv: "837ad098015d86d8c6ba7292e538df2e"
server_key_version: "encryptionkey@0"
I20260620 02:21:22.088937 12587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:22.125664 12587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:22.130314 12587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:22.130645 12587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:22.150493 12587 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33673
I20260620 02:21:22.150555 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33673 every 8 connection(s)
I20260620 02:21:22.152088 12587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:22.156250 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:22.159878 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 12587
I20260620 02:21:22.160106 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:22.160717 10141 external_mini_cluster.cc:1428] Setting key 42b9c8a35d5e8b05e547c674fd416c1c
I20260620 02:21:22.170666 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c: Bootstrap starting.
I20260620 02:21:22.174458 12654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:22.175657 12654 log.cc:824] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:21:22 dist-test-slave-9vg1 krb5kdc[12574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:22.178982 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c: No bootstrap required, opened a new log
I20260620 02:21:22.186997 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:22.163191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.9.231.126:33673)
0620 02:21:22.164024 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:22.164039 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:22.164091 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:21:22.166253 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:22.166292 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:22.166886 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:22.167423 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:22.168888 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:22.170917 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:22.172354 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:22.173455 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:22.173661 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:22.178398 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:22.178407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:22.178424 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:22.178467 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:22.182433 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:22.183265 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:22.183278 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:22.183301 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:22.183488 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:22.183930 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:22.183935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:22.183938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:22.184569 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:22.184921 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:22.185414 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 02:21:22.185874 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":329,"threads_started":1}
I20260620 02:21:22.188437 12654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f85bbc46e4c9eba94f373592ffa3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33673 } }
I20260620 02:21:22.188910 12654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:22.189057 12654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750f85bbc46e4c9eba94f373592ffa3c, State: Initialized, Role: FOLLOWER
I20260620 02:21:22.189690 12654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f85bbc46e4c9eba94f373592ffa3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33673 } }
I20260620 02:21:22.189986 12654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:22.190147 12654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:22.190366 12654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:22.192997 12654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f85bbc46e4c9eba94f373592ffa3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33673 } }
I20260620 02:21:22.193626 12654 leader_election.cc:302] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 750f85bbc46e4c9eba94f373592ffa3c; no voters:
I20260620 02:21:22.194371 12654 leader_election.cc:288] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:22.194507 12659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:22.195360 12659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [term 1 LEADER]: Becoming Leader. State: Replica: 750f85bbc46e4c9eba94f373592ffa3c, State: Running, Role: LEADER
I20260620 02:21:22.195962 12659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f85bbc46e4c9eba94f373592ffa3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33673 } }
I20260620 02:21:22.196831 12654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:22.199631 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 750f85bbc46e4c9eba94f373592ffa3c. Latest consensus state: current_term: 1 leader_uuid: "750f85bbc46e4c9eba94f373592ffa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f85bbc46e4c9eba94f373592ffa3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33673 } } }
I20260620 02:21:22.199572 12660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "750f85bbc46e4c9eba94f373592ffa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f85bbc46e4c9eba94f373592ffa3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33673 } } }
I20260620 02:21:22.200094 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:22.200094 12660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:22.208715 12668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:22.214715 12668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:22.224800 12668 catalog_manager.cc:1347] Generated new cluster ID: dac99a09a28a43b180195cea9576780e
I20260620 02:21:22.224982 12668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:22.234660 12668 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:22.236433 12668 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:22.243624 12668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 750f85bbc46e4c9eba94f373592ffa3c: Generated new TSK 0
I20260620 02:21:22.244650 12668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:22.295456 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33673
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:22.644709 12680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:22.645046 12680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:22.645147 12680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:22.652586 12680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:22.652715 12680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:22.652843 12680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:22.663010 12680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:22.664438 12680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:22.665750 12680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:22.679621 12685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:22.679555 12686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:22.679567 12688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:22.680527 12680 server_base.cc:1034] running on GCE node
I20260620 02:21:22.681201 12680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:22.682256 12680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:22.683525 12680 hybrid_clock.cc:648] HybridClock initialized: now 1781922082683455 us; error 57 us; skew 500 ppm
I20260620 02:21:22.683921 12680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:22 dist-test-slave-9vg1 krb5kdc[12574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922082, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:22.687316 12680 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:22.688977 12680 webserver.cc:460] Webserver started at http://127.9.231.65:40839/ using document root <none> and password file <none>
I20260620 02:21:22.689711 12680 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:22.689868 12680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:22.690183 12680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:22.693532 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "11c97f15f15441a29083cfd8c88fb3ed"
format_stamp: "Formatted at 2026-06-20 02:21:22 on dist-test-slave-9vg1"
server_key: "c70d3bcc747f1db5008809539984a623"
server_key_iv: "53b92e3d01bffb8bbef6b30d19277164"
server_key_version: "encryptionkey@0"
I20260620 02:21:22.694371 12680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "11c97f15f15441a29083cfd8c88fb3ed"
format_stamp: "Formatted at 2026-06-20 02:21:22 on dist-test-slave-9vg1"
server_key: "c70d3bcc747f1db5008809539984a623"
server_key_iv: "53b92e3d01bffb8bbef6b30d19277164"
server_key_version: "encryptionkey@0"
I20260620 02:21:22.700203 12680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:21:22.703809 12695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:22.705473 12680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:22.705704 12680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "11c97f15f15441a29083cfd8c88fb3ed"
format_stamp: "Formatted at 2026-06-20 02:21:22 on dist-test-slave-9vg1"
server_key: "c70d3bcc747f1db5008809539984a623"
server_key_iv: "53b92e3d01bffb8bbef6b30d19277164"
server_key_version: "encryptionkey@0"
I20260620 02:21:22.705950 12680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:22.727792 12680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:22.732421 12680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:22.732767 12680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:22.733829 12680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:22.735455 12680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:22.735565 12680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:22.735724 12680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:22.735807 12680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:22.770457 12680 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:42365
I20260620 02:21:22.770505 12807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:42365 every 8 connection(s)
I20260620 02:21:22.771929 12680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:22.776607 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 12680
I20260620 02:21:22.776821 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:22.777498 10141 external_mini_cluster.cc:1428] Setting key ed2711e65e55379f2aa22379b3ae8c09
Jun 20 02:21:22 dist-test-slave-9vg1 krb5kdc[12574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922082, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:22.793722 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:22.774733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:54993 (local address 127.9.231.126:33673)
0620 02:21:22.775258 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:22.775263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:22.776276 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 02:21:22.778457 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:22.778464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:22.778561 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:22.778693 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:22.780481 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:22.781230 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:22.782825 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:22.783354 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:22.783494 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:22.787556 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:22.787562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:22.787565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:22.787600 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:22.790346 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:22.791210 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:22.791220 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:22.791223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:22.791358 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:22.791812 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:22.791818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:22.791826 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:22.792100 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:22.792264 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:22.793035 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 02:21:22.793284 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":237,"threads_started":1}
I20260620 02:21:22.794713 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:22.775327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33673 (local address 127.9.231.65:54993)
0620 02:21:22.776100 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:22.776138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:22.778111 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:22.778950 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:22.778959 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:22.779712 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:22.780301 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:22.780320 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:22.781401 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:22.781407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:22.782667 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:22.782678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:22.783606 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:22.783615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:22.783864 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:22.784643 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:22.784675 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:22.787298 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:22.790523 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:22.790533 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:22.790553 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:22.791034 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:22.791482 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:22.791487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:22.791490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:22.791688 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:22.792232 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:22.792241 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:22.792628 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 02:21:22.792922 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 02:21:22.793496 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":275,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:21:22.796553 12808 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33673
I20260620 02:21:22.797017 12808 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:22.797894 12808 heartbeater.cc:511] Master 127.9.231.126:33673 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:21:22.800529 12619 ts_manager.cc:194] Registered new tserver with Master: 11c97f15f15441a29083cfd8c88fb3ed (127.9.231.65:42365)
I20260620 02:21:22.802035 12619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:54993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:22.822824 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33673
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:23.182829 12814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:23.183265 12814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:23.183444 12814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:23.191215 12814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:23.191390 12814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:23.191546 12814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:23.204470 12814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:23.206149 12814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:23.207851 12814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:23.223309 12820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:23.223518 12819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:23.223299 12822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:23.224442 12814 server_base.cc:1034] running on GCE node
I20260620 02:21:23.225198 12814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:23.226398 12814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:23.227682 12814 hybrid_clock.cc:648] HybridClock initialized: now 1781922083227633 us; error 45 us; skew 500 ppm
I20260620 02:21:23.228055 12814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:23 dist-test-slave-9vg1 krb5kdc[12574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:23.231581 12814 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:23.233373 12814 webserver.cc:460] Webserver started at http://127.9.231.66:35441/ using document root <none> and password file <none>
I20260620 02:21:23.234166 12814 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:23.234333 12814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:23.234656 12814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:23.237913 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "b5d65827b3bc4e1ca7810c18e6e6123e"
format_stamp: "Formatted at 2026-06-20 02:21:23 on dist-test-slave-9vg1"
server_key: "7c07e256d410768ab5b2c8362c33e80a"
server_key_iv: "24f005e6b77e139f624301d6dde150c8"
server_key_version: "encryptionkey@0"
I20260620 02:21:23.238852 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "b5d65827b3bc4e1ca7810c18e6e6123e"
format_stamp: "Formatted at 2026-06-20 02:21:23 on dist-test-slave-9vg1"
server_key: "7c07e256d410768ab5b2c8362c33e80a"
server_key_iv: "24f005e6b77e139f624301d6dde150c8"
server_key_version: "encryptionkey@0"
I20260620 02:21:23.244764 12814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:21:23.249022 12829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:23.250706 12814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 02:21:23.250968 12814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "b5d65827b3bc4e1ca7810c18e6e6123e"
format_stamp: "Formatted at 2026-06-20 02:21:23 on dist-test-slave-9vg1"
server_key: "7c07e256d410768ab5b2c8362c33e80a"
server_key_iv: "24f005e6b77e139f624301d6dde150c8"
server_key_version: "encryptionkey@0"
I20260620 02:21:23.251279 12814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:23.283735 12814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:23.288463 12814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:23.288786 12814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:23.289880 12814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:23.291460 12814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:23.291568 12814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:23.291721 12814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:23.291817 12814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:23.327617 12814 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:42519
I20260620 02:21:23.327674 12941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:42519 every 8 connection(s)
I20260620 02:21:23.329008 12814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:23.334407 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 12814
I20260620 02:21:23.334563 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:23.335127 10141 external_mini_cluster.cc:1428] Setting key 562dc87cfe3a5ca09f98e21c0619c220
Jun 20 02:21:23 dist-test-slave-9vg1 krb5kdc[12574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:23.352155 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:23.331985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:50191 (local address 127.9.231.126:33673)
0620 02:21:23.332425 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:23.332430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:23.333678 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 02:21:23.336059 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:23.336066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:23.336168 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:23.336314 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:23.338130 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.338913 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.340727 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.341175 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.341337 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:23.345482 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:23.345489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:23.345492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:23.345532 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:23.348376 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:23.349353 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:23.349358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:23.349360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:23.349503 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:23.349998 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:23.350002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:23.350004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:23.350296 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:23.350478 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:23.351430 (+ 952us) server_negotiation.cc:299] Negotiation successful
0620 02:21:23.351707 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260620 02:21:23.352864 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:23.332624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33673 (local address 127.9.231.66:50191)
0620 02:21:23.333460 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:23.333504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:23.335682 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:23.336620 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:23.336630 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:23.337250 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:23.337918 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:23.337937 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.339138 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.339142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:23.340528 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:23.340543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.341450 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.341457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:23.341729 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:23.342519 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:23.342545 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:23.345144 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:23.348589 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:23.348598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:23.348616 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:23.349105 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:23.349631 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:23.349636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:23.349640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:23.349861 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:23.350438 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:23.350447 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:23.350853 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 02:21:23.351185 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 02:21:23.351649 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:21:23.354526 12942 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33673
I20260620 02:21:23.354882 12942 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:23.355823 12942 heartbeater.cc:511] Master 127.9.231.126:33673 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
I20260620 02:21:23.357614 12619 ts_manager.cc:194] Registered new tserver with Master: b5d65827b3bc4e1ca7810c18e6e6123e (127.9.231.66:42519)
I20260620 02:21:23.358515 12619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:50191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:23.383656 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33673
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:23.725476 12948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:23.725845 12948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:23.726054 12948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:23.733366 12948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:23.733556 12948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:23.733757 12948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:23.743825 12948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:23.745287 12948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:23.746707 12948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:23.761297 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:23.761351 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:23.761407 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:23.762221 12948 server_base.cc:1034] running on GCE node
I20260620 02:21:23.762933 12948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:23.763983 12948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:23.765347 12948 hybrid_clock.cc:648] HybridClock initialized: now 1781922083765197 us; error 64 us; skew 500 ppm
I20260620 02:21:23.765744 12948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:23 dist-test-slave-9vg1 krb5kdc[12574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:23.775418 12948 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:23.777182 12948 webserver.cc:460] Webserver started at http://127.9.231.67:38113/ using document root <none> and password file <none>
I20260620 02:21:23.777953 12948 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:23.778084 12948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:23.778415 12948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:23.782135 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "c92e6dcb029b41d996472e49b27e4f79"
format_stamp: "Formatted at 2026-06-20 02:21:23 on dist-test-slave-9vg1"
server_key: "d64f0c6bec94febe41bb6f9b8b7a2af8"
server_key_iv: "136aecb828279636edd5b2eabd7ad9df"
server_key_version: "encryptionkey@0"
I20260620 02:21:23.783213 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "c92e6dcb029b41d996472e49b27e4f79"
format_stamp: "Formatted at 2026-06-20 02:21:23 on dist-test-slave-9vg1"
server_key: "d64f0c6bec94febe41bb6f9b8b7a2af8"
server_key_iv: "136aecb828279636edd5b2eabd7ad9df"
server_key_version: "encryptionkey@0"
I20260620 02:21:23.788813 12948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:21:23.792690 12963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:23.794384 12948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:23.794593 12948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "c92e6dcb029b41d996472e49b27e4f79"
format_stamp: "Formatted at 2026-06-20 02:21:23 on dist-test-slave-9vg1"
server_key: "d64f0c6bec94febe41bb6f9b8b7a2af8"
server_key_iv: "136aecb828279636edd5b2eabd7ad9df"
server_key_version: "encryptionkey@0"
I20260620 02:21:23.794842 12948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:23.805755 12808 heartbeater.cc:503] Master 127.9.231.126:33673 was elected leader, sending a full tablet report...
I20260620 02:21:23.826334 12948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:23.830746 12948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:23.831159 12948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:23.832137 12948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:23.833549 12948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:23.833621 12948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:23.833705 12948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:23.833756 12948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:23.869469 12948 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:36269
I20260620 02:21:23.869529 13075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:36269 every 8 connection(s)
I20260620 02:21:23.870805 12948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:23.875145 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 12948
I20260620 02:21:23.875332 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:23.875869 10141 external_mini_cluster.cc:1428] Setting key fc652641c6bed4946b9145b1a15000d2
Jun 20 02:21:23 dist-test-slave-9vg1 krb5kdc[12574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:23.893113 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:23.873840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:44597 (local address 127.9.231.126:33673)
0620 02:21:23.874302 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:23.874307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:23.875584 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 02:21:23.878094 (+ 2510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:23.878100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:23.878197 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:23.878315 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:23.880095 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.880887 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.882499 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.882915 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.883051 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:23.886735 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:23.886741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:23.886744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:23.886781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:23.889641 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:23.890679 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:23.890690 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:23.890692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:23.890817 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:23.891361 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:23.891364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:23.891367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:23.891730 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:23.891857 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:23.892590 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 02:21:23.892821 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
I20260620 02:21:23.894373 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:23.874467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33673 (local address 127.9.231.67:44597)
0620 02:21:23.875376 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:23.875436 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:23.877787 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:23.878581 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:23.878590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:23.879317 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:23.879924 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:23.879942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.881055 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.881067 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:23.882335 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:23.882347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:23.883186 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:23.883193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:23.883435 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:23.884120 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:23.884142 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:23.886479 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:23.889850 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:23.889860 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:23.889888 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:23.890546 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:23.890932 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:23.890937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:23.890941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:23.891233 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:23.891860 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:23.891869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:23.892242 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 02:21:23.892480 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 02:21:23.893039 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":346,"threads_started":1}
I20260620 02:21:23.896020 13076 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33673
I20260620 02:21:23.896394 13076 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:23.897212 13076 heartbeater.cc:511] Master 127.9.231.126:33673 requested a full tablet report, sending...
I20260620 02:21:23.899144 12619 ts_manager.cc:194] Registered new tserver with Master: c92e6dcb029b41d996472e49b27e4f79 (127.9.231.67:36269)
I20260620 02:21:23.900008 12619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:44597
I20260620 02:21:23.904147 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:21:23.938716 13078 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:23.936006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.9.231.126:33673)
0620 02:21:23.936253 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:23.936262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:23.936652 (+ 390us) server_negotiation.cc:407] Connection header received
0620 02:21:23.936880 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:23.936885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:23.936977 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:23.937090 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:23.938432 (+ 1342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60532: BlockingRecv error: recv got EOF from 127.0.0.1:60532 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:21:23.942288 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12680
W20260620 02:21:23.945246 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12680: Not authorized: Client connection negotiation failed: client connection to 127.9.231.65:42365: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:21:23.945600 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 12680
I20260620 02:21:23.962177 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12814
W20260620 02:21:23.964978 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12814: Not authorized: Client connection negotiation failed: client connection to 127.9.231.66:42519: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:21:23.965150 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 12814
I20260620 02:21:23.981976 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12948
W20260620 02:21:23.985258 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12948: Not authorized: Client connection negotiation failed: client connection to 127.9.231.67:36269: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:21:23.985428 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 12948
I20260620 02:21:24.002255 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 12587
I20260620 02:21:24.003796 12649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:24.115605 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 12587
2026-06-20T02:21:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5165 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:24 dist-test-slave-9vg1 krb5kdc[13094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:24 dist-test-slave-9vg1 krb5kdc[13094](info): set up 2 sockets
Jun 20 02:21:24 dist-test-slave-9vg1 krb5kdc[13094](info): commencing operation
krb5kdc: starting...
W20260620 02:21:25.257073 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:25 dist-test-slave-9vg1 krb5kdc[13094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:25.475466 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38077
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:25.821900 13107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:25.822245 13107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:25.822405 13107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:25.830233 13107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:25.830372 13107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:25.830458 13107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:25.830545 13107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:25.830631 13107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:25.841261 13107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:25.842695 13107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:25.844128 13107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:25.856822 13112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:25.856855 13113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:25.856915 13115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:25.857700 13107 server_base.cc:1034] running on GCE node
I20260620 02:21:25.858628 13107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:25.860245 13107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:25.861500 13107 hybrid_clock.cc:648] HybridClock initialized: now 1781922085861439 us; error 65 us; skew 500 ppm
I20260620 02:21:25.861896 13107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:25 dist-test-slave-9vg1 krb5kdc[13094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922085, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:25.865234 13107 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:25.867002 13107 webserver.cc:460] Webserver started at http://127.9.231.126:37721/ using document root <none> and password file <none>
I20260620 02:21:25.867762 13107 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:25.867910 13107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:25.868232 13107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:25.871435 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "0fc9576061a44cfe820ed8270ba55af1"
format_stamp: "Formatted at 2026-06-20 02:21:25 on dist-test-slave-9vg1"
server_key: "e1794ede5f210b98ec87e9b61d75f6ae"
server_key_iv: "0386895a13cf96249d8dc8952a559804"
server_key_version: "encryptionkey@0"
I20260620 02:21:25.872272 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "0fc9576061a44cfe820ed8270ba55af1"
format_stamp: "Formatted at 2026-06-20 02:21:25 on dist-test-slave-9vg1"
server_key: "e1794ede5f210b98ec87e9b61d75f6ae"
server_key_iv: "0386895a13cf96249d8dc8952a559804"
server_key_version: "encryptionkey@0"
I20260620 02:21:25.878101 13107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:21:25.881922 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:25.883464 13107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:21:25.883666 13107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "0fc9576061a44cfe820ed8270ba55af1"
format_stamp: "Formatted at 2026-06-20 02:21:25 on dist-test-slave-9vg1"
server_key: "e1794ede5f210b98ec87e9b61d75f6ae"
server_key_iv: "0386895a13cf96249d8dc8952a559804"
server_key_version: "encryptionkey@0"
I20260620 02:21:25.883908 13107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:25.901595 13107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:25.906581 13107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:25.906917 13107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:25.927517 13107 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38077
I20260620 02:21:25.927542 13173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38077 every 8 connection(s)
I20260620 02:21:25.929006 13107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:25.933349 13174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:25.936969 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13107
I20260620 02:21:25.937274 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:25.938138 10141 external_mini_cluster.cc:1428] Setting key cb5364f4750b21b2c6adc39c375fdc84
I20260620 02:21:25.948685 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1: Bootstrap starting.
I20260620 02:21:25.953284 13174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:25.954555 13174 log.cc:824] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:21:25 dist-test-slave-9vg1 krb5kdc[13094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:25.958336 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1: No bootstrap required, opened a new log
I20260620 02:21:25.965852 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:25.940513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.9.231.126:38077)
0620 02:21:25.941524 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:25.941539 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:25.941609 (+ 70us) server_negotiation.cc:407] Connection header received
0620 02:21:25.943591 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:25.943619 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:25.944172 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:25.944587 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:25.946912 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:25.948432 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:25.950414 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:25.951443 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:25.951698 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:25.956885 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:25.956895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:25.956912 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:25.956956 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:25.960749 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:25.961771 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:25.961780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:25.961788 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:25.961976 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:25.962440 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:25.962444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:25.962446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:25.963156 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:25.963464 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:25.964014 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 02:21:25.964592 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":494,"threads_started":1}
I20260620 02:21:25.968909 13174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc9576061a44cfe820ed8270ba55af1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38077 } }
I20260620 02:21:25.969389 13174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:25.969502 13174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc9576061a44cfe820ed8270ba55af1, State: Initialized, Role: FOLLOWER
I20260620 02:21:25.970253 13174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc9576061a44cfe820ed8270ba55af1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38077 } }
I20260620 02:21:25.970539 13174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:25.970738 13174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:25.970898 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:21:25.970927 13174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:25.973204 13174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc9576061a44cfe820ed8270ba55af1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38077 } }
I20260620 02:21:25.973798 13174 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc9576061a44cfe820ed8270ba55af1; no voters:
I20260620 02:21:25.974437 13174 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:25.974601 13179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:25.975860 13179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [term 1 LEADER]: Becoming Leader. State: Replica: 0fc9576061a44cfe820ed8270ba55af1, State: Running, Role: LEADER
I20260620 02:21:25.976495 13179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc9576061a44cfe820ed8270ba55af1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38077 } }
I20260620 02:21:25.976950 13174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:25.981760 13180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc9576061a44cfe820ed8270ba55af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc9576061a44cfe820ed8270ba55af1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38077 } } }
I20260620 02:21:25.982192 13180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:25.982843 13181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc9576061a44cfe820ed8270ba55af1. Latest consensus state: current_term: 1 leader_uuid: "0fc9576061a44cfe820ed8270ba55af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc9576061a44cfe820ed8270ba55af1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38077 } } }
I20260620 02:21:25.983198 13181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:25.992884 13186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:25.999774 13186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:26.025038 13186 catalog_manager.cc:1347] Generated new cluster ID: 28e5b189cd3d49b5bb6cad3686290312
I20260620 02:21:26.025239 13186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:26.038662 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:26.028425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.9.231.126:38077)
0620 02:21:26.028622 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:26.028631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:26.029135 (+ 504us) server_negotiation.cc:407] Connection header received
0620 02:21:26.029338 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:26.029345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:26.029444 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:26.029566 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:26.030869 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:26.032099 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:26.034203 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:26.034947 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:26.035177 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:26.036569 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:26.036582 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:26.036589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:26.037551 (+ 962us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:26.037642 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:26.037680 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:26.037798 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:26.037952 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:21:26.038069 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 02:21:26.045869 13186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:26.047886 13186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:26.066205 13186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc9576061a44cfe820ed8270ba55af1: Generated new TSK 0
I20260620 02:21:26.067480 13186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:21:26.084168 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13107
I20260620 02:21:26.085601 13169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:26.189051 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13107
2026-06-20T02:21:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2067 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:26 dist-test-slave-9vg1 krb5kdc[13208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:26 dist-test-slave-9vg1 krb5kdc[13208](info): set up 2 sockets
Jun 20 02:21:26 dist-test-slave-9vg1 krb5kdc[13208](info): commencing operation
krb5kdc: starting...
W20260620 02:21:27.340705 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:27 dist-test-slave-9vg1 krb5kdc[13208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:27.563369 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44547
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:27.919246 13221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:27.919605 13221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:27.919711 13221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:27.927397 13221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:27.927548 13221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:27.927601 13221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:27.927659 13221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:27.927706 13221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:27.938891 13221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:27.940455 13221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:27.941851 13221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:27.959488 13226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:27.959442 13227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:27.959533 13229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:27.961647 13221 server_base.cc:1034] running on GCE node
I20260620 02:21:27.962687 13221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:27.964385 13221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:27.965638 13221 hybrid_clock.cc:648] HybridClock initialized: now 1781922087965585 us; error 40 us; skew 500 ppm
I20260620 02:21:27.966114 13221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:27 dist-test-slave-9vg1 krb5kdc[13208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922087, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:27.969834 13221 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:27.971551 13221 webserver.cc:460] Webserver started at http://127.9.231.126:45805/ using document root <none> and password file <none>
I20260620 02:21:27.972321 13221 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:27.972486 13221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:27.972839 13221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:27.976468 13221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "91ad912477634769a923f359ad179c34"
format_stamp: "Formatted at 2026-06-20 02:21:27 on dist-test-slave-9vg1"
server_key: "ab9f8e8b3bcb0455465f3fd409524b6a"
server_key_iv: "ce865624636cd14dcd892a7bfcff9fe6"
server_key_version: "encryptionkey@0"
I20260620 02:21:27.977419 13221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "91ad912477634769a923f359ad179c34"
format_stamp: "Formatted at 2026-06-20 02:21:27 on dist-test-slave-9vg1"
server_key: "ab9f8e8b3bcb0455465f3fd409524b6a"
server_key_iv: "ce865624636cd14dcd892a7bfcff9fe6"
server_key_version: "encryptionkey@0"
I20260620 02:21:27.983541 13221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:21:27.987743 13236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:27.989547 13221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:21:27.989789 13221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "91ad912477634769a923f359ad179c34"
format_stamp: "Formatted at 2026-06-20 02:21:27 on dist-test-slave-9vg1"
server_key: "ab9f8e8b3bcb0455465f3fd409524b6a"
server_key_iv: "ce865624636cd14dcd892a7bfcff9fe6"
server_key_version: "encryptionkey@0"
I20260620 02:21:27.990056 13221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:28.005754 13221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:28.010690 13221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:28.011008 13221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:28.032223 13221 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:44547
I20260620 02:21:28.032276 13287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:44547 every 8 connection(s)
I20260620 02:21:28.033847 13221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:28.035122 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13221
I20260620 02:21:28.035444 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:28.036252 10141 external_mini_cluster.cc:1428] Setting key 81b5a4a111e12e7f6c7515fe23786140
I20260620 02:21:28.039631 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:21:28 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:28.054306 13288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34: Bootstrap starting.
I20260620 02:21:28.059095 13288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:28.060559 13288 log.cc:824] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:28.064589 13288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34: No bootstrap required, opened a new log
I20260620 02:21:28.064675 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:28.038956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.9.231.126:44547)
0620 02:21:28.039934 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:28.039948 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:28.039997 (+ 49us) server_negotiation.cc:407] Connection header received
0620 02:21:28.041647 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:28.041674 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:28.042100 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:28.042520 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:28.044446 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:28.045997 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:28.047969 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:28.048835 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:28.049008 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:28.054435 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:28.054450 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:28.054476 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:28.054538 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:28.059097 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:28.060108 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:28.060118 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:28.060127 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:28.060388 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:28.060873 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:28.060878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:28.060881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:28.061635 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:28.061911 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:28.062558 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 02:21:28.063144 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":403,"threads_started":1}
I20260620 02:21:28.072968 13288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91ad912477634769a923f359ad179c34" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44547 } }
I20260620 02:21:28.073472 13288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:28.073583 13288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ad912477634769a923f359ad179c34, State: Initialized, Role: FOLLOWER
I20260620 02:21:28.074266 13288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91ad912477634769a923f359ad179c34" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44547 } }
I20260620 02:21:28.074487 13288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:28.074580 13288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:28.074709 13288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:28.076924 13288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91ad912477634769a923f359ad179c34" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44547 } }
I20260620 02:21:28.077464 13288 leader_election.cc:302] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91ad912477634769a923f359ad179c34; no voters:
I20260620 02:21:28.078008 13288 leader_election.cc:288] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:28.078176 13293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:28.079430 13293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [term 1 LEADER]: Becoming Leader. State: Replica: 91ad912477634769a923f359ad179c34, State: Running, Role: LEADER
I20260620 02:21:28.080034 13293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91ad912477634769a923f359ad179c34" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44547 } }
I20260620 02:21:28.080475 13288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:28.082751 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91ad912477634769a923f359ad179c34. Latest consensus state: current_term: 1 leader_uuid: "91ad912477634769a923f359ad179c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91ad912477634769a923f359ad179c34" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44547 } } }
I20260620 02:21:28.083168 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:28.083729 13294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91ad912477634769a923f359ad179c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91ad912477634769a923f359ad179c34" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44547 } } }
I20260620 02:21:28.084036 13294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:28.086066 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:28.091944 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:28.102205 13301 catalog_manager.cc:1347] Generated new cluster ID: 873bd7f0679745219ef851d818943a3a
I20260620 02:21:28.102355 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:28.124568 13301 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:28.126236 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:28.136988 13301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91ad912477634769a923f359ad179c34: Generated new TSK 0
I20260620 02:21:28.138002 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:28.196342 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44547
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:28.567714 13314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:28.568085 13314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:28.568246 13314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:28.576376 13314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:28.576519 13314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:28.576642 13314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:28.587735 13314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:28.589195 13314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:28.590538 13314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:28.605640 13320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:28.606127 13322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:28.605865 13319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:28.608565 13314 server_base.cc:1034] running on GCE node
I20260620 02:21:28.609750 13314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:28.611639 13314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:28.613135 13314 hybrid_clock.cc:648] HybridClock initialized: now 1781922088613086 us; error 123 us; skew 500 ppm
I20260620 02:21:28.613903 13314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:28 dist-test-slave-9vg1 krb5kdc[13208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922088, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:28.618661 13314 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:28.620987 13314 webserver.cc:460] Webserver started at http://127.9.231.65:34341/ using document root <none> and password file <none>
I20260620 02:21:28.621965 13314 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:28.622169 13314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:28.622575 13314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:28.627501 13314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "91a6ea628086479d9b848d05973a27db"
format_stamp: "Formatted at 2026-06-20 02:21:28 on dist-test-slave-9vg1"
server_key: "d569d2baacc543631438af1adb9c46b5"
server_key_iv: "616303859275ceed4a3af5cc62347107"
server_key_version: "encryptionkey@0"
I20260620 02:21:28.628791 13314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "91a6ea628086479d9b848d05973a27db"
format_stamp: "Formatted at 2026-06-20 02:21:28 on dist-test-slave-9vg1"
server_key: "d569d2baacc543631438af1adb9c46b5"
server_key_iv: "616303859275ceed4a3af5cc62347107"
server_key_version: "encryptionkey@0"
I20260620 02:21:28.636392 13314 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 02:21:28.641022 13329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:28.642908 13314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:21:28.643183 13314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "91a6ea628086479d9b848d05973a27db"
format_stamp: "Formatted at 2026-06-20 02:21:28 on dist-test-slave-9vg1"
server_key: "d569d2baacc543631438af1adb9c46b5"
server_key_iv: "616303859275ceed4a3af5cc62347107"
server_key_version: "encryptionkey@0"
I20260620 02:21:28.643437 13314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:28.659480 13314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:28.664575 13314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:28.664997 13314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:28.666119 13314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:28.667707 13314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:28.667817 13314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:28.667984 13314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:28.668058 13314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:28.712756 13314 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:34809
I20260620 02:21:28.712817 13441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:34809 every 8 connection(s)
I20260620 02:21:28.714330 13314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:28.722047 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13314
I20260620 02:21:28.722347 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:28.723251 10141 external_mini_cluster.cc:1428] Setting key ff43f89086ef69493e128530f1b66c9f
Jun 20 02:21:28 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922088, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:28.746008 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:28.717364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:37531 (local address 127.9.231.126:44547)
0620 02:21:28.717930 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:28.717936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:28.719691 (+ 1755us) server_negotiation.cc:407] Connection header received
0620 02:21:28.722318 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:28.722325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:28.722435 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:28.722557 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:28.724915 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:28.725980 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:28.729616 (+ 3636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:28.730386 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:28.730648 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:28.736951 (+ 6303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:28.736963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:28.736972 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:28.737041 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:28.741501 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:28.742664 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:28.742674 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:28.742681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:28.742876 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:28.743579 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:28.743588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:28.743594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:28.744042 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:28.744224 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:28.745160 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 02:21:28.745482 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":273,"threads_started":1}
I20260620 02:21:28.746783 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:28.718011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44547 (local address 127.9.231.65:37531)
0620 02:21:28.719384 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:28.719462 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:28.721891 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:28.722944 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:28.722961 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:28.723848 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:28.724686 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:28.724709 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:28.726185 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:28.726191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:28.729188 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:28.729217 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:28.730806 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:28.730819 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:28.731303 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:28.732590 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:28.732639 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:28.736473 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:28.741802 (+ 5329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:28.741816 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:28.741837 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:28.742468 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:28.743036 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:28.743041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:28.743043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:28.743426 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:28.744218 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:28.744226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:28.744683 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 02:21:28.744975 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 02:21:28.745483 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":526,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:21:28.748816 13442 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44547
I20260620 02:21:28.749158 13442 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:28.750001 13442 heartbeater.cc:511] Master 127.9.231.126:44547 requested a full tablet report, sending...
I20260620 02:21:28.752590 13253 ts_manager.cc:194] Registered new tserver with Master: 91a6ea628086479d9b848d05973a27db (127.9.231.65:34809)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:21:28.754961 13253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:37531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:28.784915 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44547
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:29.164718 13448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:29.165160 13448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:29.165364 13448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:29.173339 13448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:29.173579 13448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:29.173784 13448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:29.184909 13448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:29.186528 13448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:29.187956 13448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:29.202837 13453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:29.202884 13454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:29.202919 13456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:29.203920 13448 server_base.cc:1034] running on GCE node
I20260620 02:21:29.204789 13448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:29.206187 13448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:29.209656 13448 hybrid_clock.cc:648] HybridClock initialized: now 1781922089209542 us; error 105 us; skew 500 ppm
I20260620 02:21:29.210151 13448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:29 dist-test-slave-9vg1 krb5kdc[13208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:29.219826 13448 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:29.221968 13448 webserver.cc:460] Webserver started at http://127.9.231.66:45637/ using document root <none> and password file <none>
I20260620 02:21:29.222831 13448 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:29.222954 13448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:29.223402 13448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:29.226809 13448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "6ea94b57f8b94d6a9b5a2df73b45a9e5"
format_stamp: "Formatted at 2026-06-20 02:21:29 on dist-test-slave-9vg1"
server_key: "6a9c5d532ae867c76d9dc03136fdea84"
server_key_iv: "1237d6b0e23eb52fc1411c41586bb34c"
server_key_version: "encryptionkey@0"
I20260620 02:21:29.228142 13448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "6ea94b57f8b94d6a9b5a2df73b45a9e5"
format_stamp: "Formatted at 2026-06-20 02:21:29 on dist-test-slave-9vg1"
server_key: "6a9c5d532ae867c76d9dc03136fdea84"
server_key_iv: "1237d6b0e23eb52fc1411c41586bb34c"
server_key_version: "encryptionkey@0"
I20260620 02:21:29.236781 13448 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 02:21:29.241528 13463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:29.243566 13448 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260620 02:21:29.243778 13448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "6ea94b57f8b94d6a9b5a2df73b45a9e5"
format_stamp: "Formatted at 2026-06-20 02:21:29 on dist-test-slave-9vg1"
server_key: "6a9c5d532ae867c76d9dc03136fdea84"
server_key_iv: "1237d6b0e23eb52fc1411c41586bb34c"
server_key_version: "encryptionkey@0"
I20260620 02:21:29.244048 13448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:29.261543 13448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:29.266979 13448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:29.267370 13448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:29.268497 13448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:29.270076 13448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:29.270160 13448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:29.270318 13448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:29.270399 13448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:29.312002 13448 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:34579
I20260620 02:21:29.312103 13575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:34579 every 8 connection(s)
I20260620 02:21:29.313462 13448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:29.321861 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13448
I20260620 02:21:29.322116 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:29.322556 10141 external_mini_cluster.cc:1428] Setting key 40b6777900c24ded47b7ea1b1cd7c0ae
Jun 20 02:21:29 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:29.338603 13578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:29.316869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:52653 (local address 127.9.231.126:44547)
0620 02:21:29.317429 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:29.317433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:29.318503 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 02:21:29.320574 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:29.320580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:29.320708 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:29.320838 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:29.322640 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.323571 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.325418 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.325850 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.325991 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:29.330815 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:29.330821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:29.330825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:29.330868 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:29.334071 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.335340 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.335346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.335350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.335589 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.336199 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.336203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.336206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.336591 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:29.336716 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:29.337831 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0620 02:21:29.338050 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":248,"threads_started":1}
I20260620 02:21:29.340072 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:29.317473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44547 (local address 127.9.231.66:52653)
0620 02:21:29.318293 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:29.318337 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:29.320211 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:29.321123 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:29.321132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:29.321701 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:29.322448 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:29.322469 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.323738 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.323742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:29.325249 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:29.325263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.326093 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.326100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:29.326350 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:29.327187 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:29.327212 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:29.330421 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:29.334320 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:29.334334 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:29.334360 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:29.335128 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:29.335741 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:29.335750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:29.335757 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:29.336049 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:29.336738 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:29.336749 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:29.337307 (+ 558us) client_negotiation.cc:769] Sending connection context
0620 02:21:29.337660 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 02:21:29.338353 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":333,"threads_started":1}
I20260620 02:21:29.341867 13576 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44547
I20260620 02:21:29.342243 13576 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:29.343119 13576 heartbeater.cc:511] Master 127.9.231.126:44547 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:21:29.345180 13253 ts_manager.cc:194] Registered new tserver with Master: 6ea94b57f8b94d6a9b5a2df73b45a9e5 (127.9.231.66:34579)
I20260620 02:21:29.346097 13253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:52653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:29.376035 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44547
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:29.742628 13582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:29.743029 13582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:29.743237 13582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:29.751287 13582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:29.751470 13582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:29.751650 13582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:29.758852 13442 heartbeater.cc:503] Master 127.9.231.126:44547 was elected leader, sending a full tablet report...
I20260620 02:21:29.762632 13582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:44547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:29.764171 13582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:29.765560 13582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:29.779958 13588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:29.779932 13587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:29.779968 13590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:29.781625 13582 server_base.cc:1034] running on GCE node
I20260620 02:21:29.782315 13582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:29.783846 13582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:29.785365 13582 hybrid_clock.cc:648] HybridClock initialized: now 1781922089785207 us; error 129 us; skew 500 ppm
I20260620 02:21:29.785807 13582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:29 dist-test-slave-9vg1 krb5kdc[13208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:29.789501 13582 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:29.791353 13582 webserver.cc:460] Webserver started at http://127.9.231.67:45499/ using document root <none> and password file <none>
I20260620 02:21:29.792060 13582 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:29.792166 13582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:29.792451 13582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:29.795835 13582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "edc6abbc6e79462b8366dde0e3d6dd58"
format_stamp: "Formatted at 2026-06-20 02:21:29 on dist-test-slave-9vg1"
server_key: "cdb27cd0eee63e94b0ba2e90a57c0b9c"
server_key_iv: "ba082cd870973a16fb2c4b18aacc8ab7"
server_key_version: "encryptionkey@0"
I20260620 02:21:29.796619 13582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "edc6abbc6e79462b8366dde0e3d6dd58"
format_stamp: "Formatted at 2026-06-20 02:21:29 on dist-test-slave-9vg1"
server_key: "cdb27cd0eee63e94b0ba2e90a57c0b9c"
server_key_iv: "ba082cd870973a16fb2c4b18aacc8ab7"
server_key_version: "encryptionkey@0"
I20260620 02:21:29.802162 13582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:21:29.806197 13597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:29.807920 13582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:29.808105 13582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "edc6abbc6e79462b8366dde0e3d6dd58"
format_stamp: "Formatted at 2026-06-20 02:21:29 on dist-test-slave-9vg1"
server_key: "cdb27cd0eee63e94b0ba2e90a57c0b9c"
server_key_iv: "ba082cd870973a16fb2c4b18aacc8ab7"
server_key_version: "encryptionkey@0"
I20260620 02:21:29.808341 13582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:29.823246 13582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:29.827975 13582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:29.828366 13582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:29.829378 13582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:29.830832 13582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:29.830935 13582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:29.831135 13582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:29.831226 13582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 02:21:29.867009 13582 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:40133
I20260620 02:21:29.867036 13709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:40133 every 8 connection(s)
I20260620 02:21:29.868427 13582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:29.869149 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13582
I20260620 02:21:29.869385 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:29.869980 10141 external_mini_cluster.cc:1428] Setting key e79856fac4cc14be9a9004ba8f5621b6
Jun 20 02:21:29 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:29.891393 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:29.871280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:39429 (local address 127.9.231.126:44547)
0620 02:21:29.871814 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:29.871821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:29.873260 (+ 1439us) server_negotiation.cc:407] Connection header received
0620 02:21:29.875458 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:29.875462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:29.875550 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:29.875648 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:29.877291 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.878132 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.879987 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.880506 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.880608 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:29.884953 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:29.884962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:29.884968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:29.885017 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:29.888072 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.888947 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.888951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.888954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.889078 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.889584 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.889587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.889589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.889871 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:29.889989 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:29.890831 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 02:21:29.891005 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":278,"threads_started":1}
I20260620 02:21:29.892421 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:29.871977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:44547 (local address 127.9.231.67:39429)
0620 02:21:29.873066 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:29.873128 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:29.875119 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:29.875890 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:29.875899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:29.876443 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:29.877051 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:29.877069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.878335 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.878341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:29.879797 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:29.879812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.880737 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.880743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:29.880972 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:29.881806 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:29.881831 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:29.884402 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:29.888252 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:29.888261 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:29.888279 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:29.888811 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:29.889234 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:29.889238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:29.889241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:29.889469 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:29.890008 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:29.890015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:29.890441 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 02:21:29.890686 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 02:21:29.891197 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":480,"threads_started":1}
I20260620 02:21:29.893998 13710 heartbeater.cc:347] Connected to a master server at 127.9.231.126:44547
I20260620 02:21:29.894414 13710 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:29.895310 13710 heartbeater.cc:511] Master 127.9.231.126:44547 requested a full tablet report, sending...
I20260620 02:21:29.896885 13253 ts_manager.cc:194] Registered new tserver with Master: edc6abbc6e79462b8366dde0e3d6dd58 (127.9.231.67:40133)
I20260620 02:21:29.897908 13253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:39429
I20260620 02:21:29.900085 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:21:29 dist-test-slave-9vg1 krb5kdc[13208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 02:21:29 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:29.956135 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:29.939562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.9.231.126:44547)
0620 02:21:29.939939 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:29.939948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:29.940373 (+ 425us) server_negotiation.cc:407] Connection header received
0620 02:21:29.940605 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:29.940609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:29.940677 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:29.940789 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:29.942193 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.942936 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.944258 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.944720 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.945056 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:29.949651 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:29.949657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:29.949661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:29.949714 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:29.952808 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.953799 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.953805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.953808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.953961 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.954562 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.954567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.954569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.954951 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:29.955240 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:29.955634 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 02:21:29.955860 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 02:21:29.956981 13253 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55528
I20260620 02:21:29.959945 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13314
Jun 20 02:21:29 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:29.978672 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:29.961136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.9.231.65:34809)
0620 02:21:29.961834 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:29.961841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:29.961868 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:21:29.961927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:29.961934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:29.962155 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:29.962302 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:29.964597 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.965706 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.967129 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:29.967778 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:29.967904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:29.972310 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:29.972328 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:29.972346 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:29.972390 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:29.975546 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.976337 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.976345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.976347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.976486 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:29.976943 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:29.976948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:29.976950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:29.977331 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:29.977674 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:29.978001 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 02:21:29.978281 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":386,"threads_started":1}
W20260620 02:21:29.980134 13437 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45560
W20260620 02:21:29.981462 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:29.981688 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13314
I20260620 02:21:29.999398 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13448
Jun 20 02:21:30 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:21:30.018710 13723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:30.000439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.9.231.66:34579)
0620 02:21:30.001048 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:30.001056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:30.001082 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:21:30.001176 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:30.001185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:30.001424 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:30.001563 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:30.003006 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:30.004203 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:30.005641 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:30.006251 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:30.006357 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:30.010925 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:30.010943 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:30.010960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:30.011003 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:30.014774 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:30.016090 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:30.016111 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:30.016113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:30.016299 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:30.016840 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:30.016844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:30.016847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:30.017247 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:30.017689 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:30.018028 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 02:21:30.018320 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":326,"threads_started":1}
W20260620 02:21:30.020232 13571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52958
W20260620 02:21:30.021730 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13448: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:30.021960 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13448
I20260620 02:21:30.041256 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13582
Jun 20 02:21:30 dist-test-slave-9vg1 krb5kdc[13208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922089, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:21:30.060252 13724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:30.042355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.9.231.67:40133)
0620 02:21:30.043177 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:30.043184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:30.043215 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:21:30.043276 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:30.043282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:30.043541 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:30.043727 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:30.045211 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:30.046304 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:30.047818 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:30.048427 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:30.048583 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:30.053506 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:30.053522 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:30.053540 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:30.053586 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:30.056937 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:30.057871 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:30.057879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:30.057882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:30.058031 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:30.058458 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:30.058463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:30.058466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:30.058855 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:30.059241 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:30.059565 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 02:21:30.059875 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":431,"threads_started":1}
W20260620 02:21:30.061738 13705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40948
W20260620 02:21:30.062973 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:21:30.063184 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13582
I20260620 02:21:30.079475 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13221
I20260620 02:21:30.080828 13283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:30.189463 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13221
2026-06-20T02:21:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4005 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:30 dist-test-slave-9vg1 krb5kdc[13728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:30 dist-test-slave-9vg1 krb5kdc[13728](info): set up 2 sockets
Jun 20 02:21:30 dist-test-slave-9vg1 krb5kdc[13728](info): commencing operation
krb5kdc: starting...
W20260620 02:21:31.340737 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:31 dist-test-slave-9vg1 krb5kdc[13728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:31.555420 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34809
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:31.897509 13741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:31.897825 13741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:31.897971 13741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:31.906006 13741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:31.906140 13741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:31.906244 13741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:31.906366 13741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:31.906447 13741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:31.917740 13741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:31.919334 13741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:31.920783 13741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:31.934686 13747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:31.934758 13746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:31.934620 13749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:31.935508 13741 server_base.cc:1034] running on GCE node
I20260620 02:21:31.936443 13741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:31.938208 13741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:31.939474 13741 hybrid_clock.cc:648] HybridClock initialized: now 1781922091939398 us; error 64 us; skew 500 ppm
I20260620 02:21:31.940047 13741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:31 dist-test-slave-9vg1 krb5kdc[13728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922091, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:31.943727 13741 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:31.945675 13741 webserver.cc:460] Webserver started at http://127.9.231.126:40715/ using document root <none> and password file <none>
I20260620 02:21:31.946513 13741 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:31.946659 13741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:31.947028 13741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:31.950374 13741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "ea9ce962faf84dd19deabac3cbe47381"
format_stamp: "Formatted at 2026-06-20 02:21:31 on dist-test-slave-9vg1"
server_key: "47e86cf4ff732bfd86e50e34a8f9e798"
server_key_iv: "7c0e52e80488a2a34aca4e9529c480b0"
server_key_version: "encryptionkey@0"
I20260620 02:21:31.951378 13741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "ea9ce962faf84dd19deabac3cbe47381"
format_stamp: "Formatted at 2026-06-20 02:21:31 on dist-test-slave-9vg1"
server_key: "47e86cf4ff732bfd86e50e34a8f9e798"
server_key_iv: "7c0e52e80488a2a34aca4e9529c480b0"
server_key_version: "encryptionkey@0"
I20260620 02:21:31.957993 13741 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:21:31.962494 13756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:31.964126 13741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:31.964398 13741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "ea9ce962faf84dd19deabac3cbe47381"
format_stamp: "Formatted at 2026-06-20 02:21:31 on dist-test-slave-9vg1"
server_key: "47e86cf4ff732bfd86e50e34a8f9e798"
server_key_iv: "7c0e52e80488a2a34aca4e9529c480b0"
server_key_version: "encryptionkey@0"
I20260620 02:21:31.964645 13741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:31.977718 13741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:31.983037 13741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:31.983431 13741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:32.003821 13741 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34809
I20260620 02:21:32.003854 13807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34809 every 8 connection(s)
I20260620 02:21:32.005494 13741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:32.007149 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13741
I20260620 02:21:32.007360 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:32.008167 10141 external_mini_cluster.cc:1428] Setting key 6dc246ded55901d7accf241e82d3cdb2
I20260620 02:21:32.010766 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:21:32 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:32.025221 13808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381: Bootstrap starting.
I20260620 02:21:32.029721 13808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:32.031107 13808 log.cc:824] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:32.034205 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:32.010385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.9.231.126:34809)
0620 02:21:32.011277 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:32.011290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:32.011331 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:21:32.013031 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:32.013052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:32.013484 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:32.013874 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:32.015889 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:32.017402 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:32.019225 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:32.019955 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:32.020100 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:32.024959 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:32.024972 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:32.024995 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:32.025049 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:32.029325 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:32.030310 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:32.030319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:32.030329 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:32.030543 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:32.031002 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:32.031007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:32.031014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:32.031894 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:32.032250 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:32.032682 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 02:21:32.033150 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":432,"threads_started":1}
I20260620 02:21:32.034439 13808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381: No bootstrap required, opened a new log
I20260620 02:21:32.044075 13808 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ce962faf84dd19deabac3cbe47381" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34809 } }
I20260620 02:21:32.044780 13808 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:32.044961 13808 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ce962faf84dd19deabac3cbe47381, State: Initialized, Role: FOLLOWER
I20260620 02:21:32.045781 13808 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ce962faf84dd19deabac3cbe47381" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34809 } }
I20260620 02:21:32.046108 13808 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:32.046360 13808 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:32.046554 13808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:32.049014 13808 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ce962faf84dd19deabac3cbe47381" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34809 } }
I20260620 02:21:32.049608 13808 leader_election.cc:302] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9ce962faf84dd19deabac3cbe47381; no voters:
I20260620 02:21:32.050374 13808 leader_election.cc:288] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:32.050491 13813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:32.051296 13813 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [term 1 LEADER]: Becoming Leader. State: Replica: ea9ce962faf84dd19deabac3cbe47381, State: Running, Role: LEADER
I20260620 02:21:32.051986 13813 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ce962faf84dd19deabac3cbe47381" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34809 } }
I20260620 02:21:32.052708 13808 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:32.055444 13815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9ce962faf84dd19deabac3cbe47381. Latest consensus state: current_term: 1 leader_uuid: "ea9ce962faf84dd19deabac3cbe47381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ce962faf84dd19deabac3cbe47381" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34809 } } }
I20260620 02:21:32.056012 13815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:32.056073 13814 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9ce962faf84dd19deabac3cbe47381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ce962faf84dd19deabac3cbe47381" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34809 } } }
I20260620 02:21:32.056329 13814 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381 [sys.catalog]: This master's current role is: LEADER
W20260620 02:21:32.064281 13829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:21:32.064431 13829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:21:32.064596 13822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:32.066915 13822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:32.076519 13822 catalog_manager.cc:1347] Generated new cluster ID: 385ddb0ae7284b298223d549829ede36
I20260620 02:21:32.076651 13822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:32.095117 13822 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:32.096622 13822 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:32.120955 13822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9ce962faf84dd19deabac3cbe47381: Generated new TSK 0
I20260620 02:21:32.122030 13822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:32.183313 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34809
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:32.557704 13834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:32.557997 13834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:32.558169 13834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:32.565893 13834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:32.566051 13834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:32.566288 13834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:32.577189 13834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:32.578609 13834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:32.580040 13834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:32.594972 13839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:32.595098 13842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:32.595103 13840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:32.595916 13834 server_base.cc:1034] running on GCE node
I20260620 02:21:32.596647 13834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:32.597892 13834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:32.599188 13834 hybrid_clock.cc:648] HybridClock initialized: now 1781922092599130 us; error 56 us; skew 500 ppm
I20260620 02:21:32.599560 13834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:32 dist-test-slave-9vg1 krb5kdc[13728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922092, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:32.603118 13834 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:32.604990 13834 webserver.cc:460] Webserver started at http://127.9.231.65:37733/ using document root <none> and password file <none>
I20260620 02:21:32.605844 13834 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:32.606014 13834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:32.606359 13834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:32.609707 13834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "e49b6b0f2914420894e227573528d329"
format_stamp: "Formatted at 2026-06-20 02:21:32 on dist-test-slave-9vg1"
server_key: "37d11490de630a8fc8aa64219cea988c"
server_key_iv: "0ff6df554d16280ebc5e95b0d4800d76"
server_key_version: "encryptionkey@0"
I20260620 02:21:32.610632 13834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "e49b6b0f2914420894e227573528d329"
format_stamp: "Formatted at 2026-06-20 02:21:32 on dist-test-slave-9vg1"
server_key: "37d11490de630a8fc8aa64219cea988c"
server_key_iv: "0ff6df554d16280ebc5e95b0d4800d76"
server_key_version: "encryptionkey@0"
I20260620 02:21:32.616993 13834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260620 02:21:32.621192 13849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:32.622715 13834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:21:32.622931 13834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "e49b6b0f2914420894e227573528d329"
format_stamp: "Formatted at 2026-06-20 02:21:32 on dist-test-slave-9vg1"
server_key: "37d11490de630a8fc8aa64219cea988c"
server_key_iv: "0ff6df554d16280ebc5e95b0d4800d76"
server_key_version: "encryptionkey@0"
I20260620 02:21:32.623284 13834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:32.667948 13834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:32.673077 13834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:32.673440 13834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:32.674439 13834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:32.675904 13834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:32.676015 13834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:32.676173 13834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:32.676247 13834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:32.711676 13834 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:36647
I20260620 02:21:32.711771 13961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:36647 every 8 connection(s)
I20260620 02:21:32.713073 13834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:32.719997 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13834
I20260620 02:21:32.720353 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:32.721017 10141 external_mini_cluster.cc:1428] Setting key 1dfb3ebaf44920a5e2804e0bb6c0b2a6
Jun 20 02:21:32 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922092, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:32.735436 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:32.715735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:40843 (local address 127.9.231.126:34809)
0620 02:21:32.716229 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:32.716235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:32.717087 (+ 852us) server_negotiation.cc:407] Connection header received
0620 02:21:32.719018 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:32.719023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:32.719156 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:32.719291 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:32.720995 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:32.721852 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:32.723958 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:32.724371 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:32.724515 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:32.728754 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:32.728760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:32.728763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:32.728797 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:32.731505 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:32.732561 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:32.732566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:32.732569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:32.732703 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:32.733251 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:32.733255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:32.733257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:32.733643 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:32.733770 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:32.734758 (+ 988us) server_negotiation.cc:299] Negotiation successful
0620 02:21:32.734971 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":260,"threads_started":1}
I20260620 02:21:32.736893 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:32.716196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34809 (local address 127.9.231.65:40843)
0620 02:21:32.716895 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:32.716945 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:32.718720 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:32.719560 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:32.719569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:32.720099 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:32.720807 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:32.720827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:32.722024 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:32.722028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:32.723781 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:32.723795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:32.724607 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:32.724614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:32.724898 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:32.725607 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:32.725631 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:32.728497 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:32.731670 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:32.731683 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:32.731706 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:32.732391 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:32.732843 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:32.732851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:32.732857 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:32.733128 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:32.733782 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:32.733794 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:32.734304 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 02:21:32.734637 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 02:21:32.735306 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260620 02:21:32.738713 13962 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34809
I20260620 02:21:32.739099 13962 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:32.739887 13962 heartbeater.cc:511] Master 127.9.231.126:34809 requested a full tablet report, sending...
I20260620 02:21:32.742156 13773 ts_manager.cc:194] Registered new tserver with Master: e49b6b0f2914420894e227573528d329 (127.9.231.65:36647)
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:21:32.743866 13773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:40843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:32.771659 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34809
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:33.143054 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:33.143519 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:33.143702 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:33.151705 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:33.151829 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:33.151943 13968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:33.162338 13968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:33.163743 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:33.165077 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:33.179633 13973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:33.179596 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:33.179680 13974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:33.185298 13968 server_base.cc:1034] running on GCE node
I20260620 02:21:33.186053 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:33.187160 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:33.188416 13968 hybrid_clock.cc:648] HybridClock initialized: now 1781922093188347 us; error 65 us; skew 500 ppm
I20260620 02:21:33.188776 13968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:33 dist-test-slave-9vg1 krb5kdc[13728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:33.192082 13968 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:33.193715 13968 webserver.cc:460] Webserver started at http://127.9.231.66:40055/ using document root <none> and password file <none>
I20260620 02:21:33.194428 13968 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:33.194530 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:33.194784 13968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:33.197783 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "f716dea02a1d471891d8ec9da0e7229f"
format_stamp: "Formatted at 2026-06-20 02:21:33 on dist-test-slave-9vg1"
server_key: "501ef7f6346fe2379f66374e63a543b0"
server_key_iv: "642173898b90cdb99b4d2ac854437199"
server_key_version: "encryptionkey@0"
I20260620 02:21:33.198510 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "f716dea02a1d471891d8ec9da0e7229f"
format_stamp: "Formatted at 2026-06-20 02:21:33 on dist-test-slave-9vg1"
server_key: "501ef7f6346fe2379f66374e63a543b0"
server_key_iv: "642173898b90cdb99b4d2ac854437199"
server_key_version: "encryptionkey@0"
I20260620 02:21:33.203981 13968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 02:21:33.207861 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:33.209403 13968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:33.209609 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "f716dea02a1d471891d8ec9da0e7229f"
format_stamp: "Formatted at 2026-06-20 02:21:33 on dist-test-slave-9vg1"
server_key: "501ef7f6346fe2379f66374e63a543b0"
server_key_iv: "642173898b90cdb99b4d2ac854437199"
server_key_version: "encryptionkey@0"
I20260620 02:21:33.209856 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:33.231794 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:33.236874 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:33.237242 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:33.238677 13968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:33.240430 13968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:33.240514 13968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:33.240686 13968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:33.240756 13968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:33.278002 13968 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:35051
I20260620 02:21:33.278100 14095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:35051 every 8 connection(s)
I20260620 02:21:33.279477 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:33.284396 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 13968
I20260620 02:21:33.284631 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:33.285269 10141 external_mini_cluster.cc:1428] Setting key 7a34dddc1e45c81db54c1d64498f699a
Jun 20 02:21:33 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:33.305979 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:33.282385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:42537 (local address 127.9.231.126:34809)
0620 02:21:33.283022 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:33.283030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:33.284108 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 02:21:33.286924 (+ 2816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:33.286929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:33.286999 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:33.287189 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:33.289669 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.290541 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.293357 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.293809 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.293936 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:33.297739 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:33.297749 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:33.297757 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:33.297806 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:33.301692 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.302764 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.302773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.302779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.302935 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.303756 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.303764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.303771 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.304233 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:33.304413 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:33.305279 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 02:21:33.305553 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":279,"threads_started":1}
I20260620 02:21:33.306771 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:33.282964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34809 (local address 127.9.231.66:42537)
0620 02:21:33.283850 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:33.283937 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:33.286501 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:33.287496 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:33.287511 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:33.288326 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:33.289419 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:33.289453 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.290733 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.290742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:33.293092 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:33.293113 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.294089 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.294099 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:33.294429 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:33.295230 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:33.295259 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:33.297461 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:33.301903 (+ 4442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:33.301911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:33.301928 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:33.302581 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:33.303144 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:33.303149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:33.303171 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:33.303593 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:33.304421 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:33.304428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:33.304832 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 02:21:33.305102 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 02:21:33.305559 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:21:33.308466 14096 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34809
I20260620 02:21:33.308861 14096 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:33.309667 14096 heartbeater.cc:511] Master 127.9.231.126:34809 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
I20260620 02:21:33.311444 13773 ts_manager.cc:194] Registered new tserver with Master: f716dea02a1d471891d8ec9da0e7229f (127.9.231.66:35051)
I20260620 02:21:33.312323 13773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:42537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:33.339246 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34809
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:33.685992 14102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:33.686319 14102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:33.686427 14102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:33.693925 14102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:33.694058 14102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:33.694173 14102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:33.704298 14102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:33.705652 14102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:33.706957 14102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:33.721561 14107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:33.721740 14110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:33.721783 14108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:33.723135 14102 server_base.cc:1034] running on GCE node
I20260620 02:21:33.724035 14102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:33.725078 14102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:33.726310 14102 hybrid_clock.cc:648] HybridClock initialized: now 1781922093726227 us; error 71 us; skew 500 ppm
I20260620 02:21:33.726711 14102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:33 dist-test-slave-9vg1 krb5kdc[13728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:33.730252 14102 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:33.731998 14102 webserver.cc:460] Webserver started at http://127.9.231.67:38333/ using document root <none> and password file <none>
I20260620 02:21:33.732774 14102 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:33.732918 14102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:33.733276 14102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:33.736492 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "d326a6edfe4a40eb8a294f22ae36d43b"
format_stamp: "Formatted at 2026-06-20 02:21:33 on dist-test-slave-9vg1"
server_key: "6adb114c4703067839f3f077e333dd1b"
server_key_iv: "323b623710909353c10f6d6eadc0084c"
server_key_version: "encryptionkey@0"
I20260620 02:21:33.737306 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "d326a6edfe4a40eb8a294f22ae36d43b"
format_stamp: "Formatted at 2026-06-20 02:21:33 on dist-test-slave-9vg1"
server_key: "6adb114c4703067839f3f077e333dd1b"
server_key_iv: "323b623710909353c10f6d6eadc0084c"
server_key_version: "encryptionkey@0"
I20260620 02:21:33.743237 14102 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:21:33.747151 14117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:33.748411 13962 heartbeater.cc:503] Master 127.9.231.126:34809 was elected leader, sending a full tablet report...
I20260620 02:21:33.749069 14102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:21:33.749262 14102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "d326a6edfe4a40eb8a294f22ae36d43b"
format_stamp: "Formatted at 2026-06-20 02:21:33 on dist-test-slave-9vg1"
server_key: "6adb114c4703067839f3f077e333dd1b"
server_key_iv: "323b623710909353c10f6d6eadc0084c"
server_key_version: "encryptionkey@0"
I20260620 02:21:33.749498 14102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:33.764878 14102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:33.769714 14102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:33.770023 14102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:33.771042 14102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:33.772540 14102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:33.772614 14102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:33.772771 14102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:33.772835 14102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:33.807754 14102 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:39249
I20260620 02:21:33.807757 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:39249 every 8 connection(s)
I20260620 02:21:33.809168 14102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:33.811019 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 14102
I20260620 02:21:33.811250 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:33.811838 10141 external_mini_cluster.cc:1428] Setting key 40f13b666d292c5213d9da5dc919f731
Jun 20 02:21:33 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:33.830897 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:33.812019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:54335 (local address 127.9.231.126:34809)
0620 02:21:33.812512 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:33.812517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:33.813635 (+ 1118us) server_negotiation.cc:407] Connection header received
0620 02:21:33.815985 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:33.815990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:33.816082 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:33.816199 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:33.817785 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.818600 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.820249 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.820657 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.820798 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:33.824403 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:33.824408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:33.824411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:33.824444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:33.827508 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.828435 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.828441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.828446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.828568 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.829023 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.829028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.829032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.829306 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:33.829472 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:33.830236 (+ 764us) server_negotiation.cc:299] Negotiation successful
0620 02:21:33.830493 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
I20260620 02:21:33.831911 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:33.812655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34809 (local address 127.9.231.67:54335)
0620 02:21:33.813455 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:33.813510 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:33.815701 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:33.816458 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:33.816468 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:33.817008 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:33.817613 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:33.817630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.818769 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.818773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:33.820098 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:33.820111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.820905 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.820913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:33.821146 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:33.821862 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:33.821885 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:33.824156 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:33.827677 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:33.827746 (+ 69us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:33.827768 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:33.828305 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:33.828685 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:33.828690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:33.828693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:33.828901 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:33.829459 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:33.829467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:33.829845 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 02:21:33.830124 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 02:21:33.830698 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":434,"threads_started":1}
I20260620 02:21:33.833304 14230 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34809
I20260620 02:21:33.833600 14230 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:33.834311 14230 heartbeater.cc:511] Master 127.9.231.126:34809 requested a full tablet report, sending...
I20260620 02:21:33.836082 13773 ts_manager.cc:194] Registered new tserver with Master: d326a6edfe4a40eb8a294f22ae36d43b (127.9.231.67:39249)
I20260620 02:21:33.836946 13773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:54335
I20260620 02:21:33.841313 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:21:33 dist-test-slave-9vg1 krb5kdc[13728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:21:33 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:33.894261 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:33.877926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.9.231.65:36647)
0620 02:21:33.878613 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:33.878620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:33.878646 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:21:33.878704 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:33.878710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:33.878936 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:33.879132 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:33.880680 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.881709 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.883017 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.883658 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.883813 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:33.888179 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:33.888194 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:33.888210 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:33.888256 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:33.891321 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.892036 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.892044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.892046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.892173 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.892557 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.892564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.892567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.892994 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:33.893322 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:33.893616 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 02:21:33.893903 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":343,"threads_started":1}
Jun 20 02:21:33 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:33.914700 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:33.899180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.9.231.126:34809)
0620 02:21:33.899413 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:33.899422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:33.899539 (+ 117us) server_negotiation.cc:407] Connection header received
0620 02:21:33.899681 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:33.899685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:33.899783 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:33.899869 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:33.901220 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.902030 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.903415 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.903836 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.903945 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:33.908694 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:33.908701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:33.908705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:33.908751 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:33.911841 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.912583 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.912587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.912589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.912735 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.913054 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.913057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.913059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.913478 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:33.913692 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:33.914013 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:21:33.914396 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 02:21:33.915416 13773 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47272
I20260620 02:21:33.916736 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13834
I20260620 02:21:33.930819 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:33.917935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.9.231.65:36647)
0620 02:21:33.918212 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:33.918223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:33.918352 (+ 129us) server_negotiation.cc:407] Connection header received
0620 02:21:33.918483 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:33.918487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:33.918571 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:33.918661 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:33.920188 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.920899 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.922289 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:33.922711 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:33.922823 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:33.925356 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:33.925361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:33.925364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:33.925397 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:33.928485 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.929129 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.929133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.929146 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.929319 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:33.929640 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:33.929643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:33.929645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:33.929883 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:33.930053 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:33.930310 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 02:21:33.930473 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:21:33.931533 13957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:34.018393 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13834
I20260620 02:21:34.036367 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13968
Jun 20 02:21:34 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:21:34.054992 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:34.037350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.9.231.66:35051)
0620 02:21:34.037957 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:34.037964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:34.037990 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:21:34.038050 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:34.038056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:34.038277 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:34.038420 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:34.039763 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:34.040849 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:34.042305 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:34.042928 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:34.043042 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:34.047969 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:34.047988 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:34.048005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:34.048062 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:34.051478 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:34.052335 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:34.052343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:34.052346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:34.052563 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:34.053053 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:34.053057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:34.053059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:34.053654 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:34.053975 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:34.054398 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 02:21:34.054674 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":326,"threads_started":1}
I20260620 02:21:34.056756 14091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:34.140937 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13968
I20260620 02:21:34.160590 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 14102
Jun 20 02:21:34 dist-test-slave-9vg1 krb5kdc[13728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:21:34.178547 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:34.161733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.9.231.67:39249)
0620 02:21:34.162359 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:34.162366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:34.162396 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:21:34.162459 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:34.162465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:34.162703 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:34.162889 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:34.164194 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:34.165281 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:34.166601 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:34.167305 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:34.167452 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:34.172197 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:34.172210 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:34.172228 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:34.172275 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:34.175642 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:34.176435 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:34.176444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:34.176446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:34.176600 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:34.176998 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:34.177002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:34.177004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:34.177419 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:34.177719 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:34.178017 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 02:21:34.178234 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":318,"threads_started":1}
I20260620 02:21:34.179998 14225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:34.269034 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 14102
I20260620 02:21:34.287439 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 13741
I20260620 02:21:34.288736 13803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:34.398813 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 13741
2026-06-20T02:21:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4209 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:34 dist-test-slave-9vg1 krb5kdc[14250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:34 dist-test-slave-9vg1 krb5kdc[14250](info): set up 2 sockets
Jun 20 02:21:34 dist-test-slave-9vg1 krb5kdc[14250](info): commencing operation
krb5kdc: starting...
W20260620 02:21:35.528842 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:35 dist-test-slave-9vg1 krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:35.751408 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40015
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:36.125806 14263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:36.126176 14263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:36.126369 14263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:36.135385 14263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:36.135524 14263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:36.135634 14263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:36.135737 14263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:36.135808 14263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:36.147025 14263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:36.148486 14263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:36.149866 14263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:36.163620 14268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:36.163589 14271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:36.163687 14269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:36.165109 14263 server_base.cc:1034] running on GCE node
I20260620 02:21:36.165979 14263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:36.167649 14263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:36.168895 14263 hybrid_clock.cc:648] HybridClock initialized: now 1781922096168847 us; error 38 us; skew 500 ppm
I20260620 02:21:36.169332 14263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:36 dist-test-slave-9vg1 krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922096, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:36.173069 14263 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:36.174512 14263 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:36.174686 14263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:36.175151 14263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:36.178792 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "14532d8c158d42eba1339ac400099148"
format_stamp: "Formatted at 2026-06-20 02:21:36 on dist-test-slave-9vg1"
server_key: "3244cc61b900342e565df2049a4252c2"
server_key_iv: "b9c36744791cc0b83139ff98c31e62aa"
server_key_version: "encryptionkey@0"
I20260620 02:21:36.179921 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "14532d8c158d42eba1339ac400099148"
format_stamp: "Formatted at 2026-06-20 02:21:36 on dist-test-slave-9vg1"
server_key: "3244cc61b900342e565df2049a4252c2"
server_key_iv: "b9c36744791cc0b83139ff98c31e62aa"
server_key_version: "encryptionkey@0"
I20260620 02:21:36.186425 14263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:21:36.190819 14277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:36.192688 14263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:21:36.192960 14263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "14532d8c158d42eba1339ac400099148"
format_stamp: "Formatted at 2026-06-20 02:21:36 on dist-test-slave-9vg1"
server_key: "3244cc61b900342e565df2049a4252c2"
server_key_iv: "b9c36744791cc0b83139ff98c31e62aa"
server_key_version: "encryptionkey@0"
I20260620 02:21:36.193249 14263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:36.212520 14263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:36.217517 14263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:36.217849 14263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:36.238831 14263 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:40015
I20260620 02:21:36.238849 14328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:40015 every 8 connection(s)
I20260620 02:21:36.239964 14263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:36.244129 14329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:36.245262 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 14263
I20260620 02:21:36.245497 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:36.246316 10141 external_mini_cluster.cc:1428] Setting key 186ee64b932a1e047c77d82eb06878e8
I20260620 02:21:36.261507 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148: Bootstrap starting.
Jun 20 02:21:36 dist-test-slave-9vg1 krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:36.265781 14329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:36.267422 14329 log.cc:824] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:36.271390 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148: No bootstrap required, opened a new log
I20260620 02:21:36.274858 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:36.248760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.9.231.126:40015)
0620 02:21:36.249743 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:36.249760 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:36.249808 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:21:36.251635 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:36.251663 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:36.252173 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:36.252575 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:36.254795 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:36.256402 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:36.258391 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:36.259284 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:36.259453 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:36.264916 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:36.264931 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:36.264955 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:36.265013 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:36.269315 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:36.270306 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:36.270314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:36.270324 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:36.270557 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:36.271048 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:36.271053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:36.271091 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:36.271905 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:36.272251 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:36.272762 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 02:21:36.273268 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":731,"thread_start_us":457,"threads_started":1}
I20260620 02:21:36.281705 14329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14532d8c158d42eba1339ac400099148" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40015 } }
I20260620 02:21:36.282124 14329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:36.282212 14329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14532d8c158d42eba1339ac400099148, State: Initialized, Role: FOLLOWER
I20260620 02:21:36.282835 14329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14532d8c158d42eba1339ac400099148" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40015 } }
I20260620 02:21:36.283133 14329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:36.283308 14329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:36.283504 14329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:36.286238 14329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14532d8c158d42eba1339ac400099148" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40015 } }
I20260620 02:21:36.286878 14329 leader_election.cc:302] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14532d8c158d42eba1339ac400099148; no voters:
I20260620 02:21:36.287554 14329 leader_election.cc:288] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:36.287813 14334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:36.288985 14334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [term 1 LEADER]: Becoming Leader. State: Replica: 14532d8c158d42eba1339ac400099148, State: Running, Role: LEADER
I20260620 02:21:36.290325 14329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:36.289994 14334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14532d8c158d42eba1339ac400099148" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40015 } }
I20260620 02:21:36.294929 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14532d8c158d42eba1339ac400099148. Latest consensus state: current_term: 1 leader_uuid: "14532d8c158d42eba1339ac400099148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14532d8c158d42eba1339ac400099148" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40015 } } }
I20260620 02:21:36.295277 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14532d8c158d42eba1339ac400099148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14532d8c158d42eba1339ac400099148" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40015 } } }
I20260620 02:21:36.295617 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:36.295614 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:36.298780 14343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:36.306107 14343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:36.316848 14343 catalog_manager.cc:1347] Generated new cluster ID: 4116af3279034650b9d4b5a27f2ce563
I20260620 02:21:36.317020 14343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:36.336280 14343 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:36.337882 14343 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:36.344977 14343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148: Generated new TSK 0
I20260620 02:21:36.346261 14343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:36.405128 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40015
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:36.771755 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:36.772082 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:36.772194 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:36.780109 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:36.780323 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:36.780452 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:36.791175 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:36.792673 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:36.794064 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:36.811507 14360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:36.811548 14363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:36.811466 14361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:36.812628 14355 server_base.cc:1034] running on GCE node
I20260620 02:21:36.813428 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:36.814496 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:36.815744 14355 hybrid_clock.cc:648] HybridClock initialized: now 1781922096815667 us; error 66 us; skew 500 ppm
I20260620 02:21:36.816146 14355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:36 dist-test-slave-9vg1 krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922096, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:36.819908 14355 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:36.821218 14355 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:36.821327 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:36.821607 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:36.824965 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "97ffde688585457ab4d3726fc239b478"
format_stamp: "Formatted at 2026-06-20 02:21:36 on dist-test-slave-9vg1"
server_key: "a6f152ab1f39e2d7fa183ccf925514a5"
server_key_iv: "330e93dfe74f2bc13653bf73baa509de"
server_key_version: "encryptionkey@0"
I20260620 02:21:36.825870 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "97ffde688585457ab4d3726fc239b478"
format_stamp: "Formatted at 2026-06-20 02:21:36 on dist-test-slave-9vg1"
server_key: "a6f152ab1f39e2d7fa183ccf925514a5"
server_key_iv: "330e93dfe74f2bc13653bf73baa509de"
server_key_version: "encryptionkey@0"
I20260620 02:21:36.831828 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:21:36.836086 14369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:36.837970 14355 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:21:36.838253 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "97ffde688585457ab4d3726fc239b478"
format_stamp: "Formatted at 2026-06-20 02:21:36 on dist-test-slave-9vg1"
server_key: "a6f152ab1f39e2d7fa183ccf925514a5"
server_key_iv: "330e93dfe74f2bc13653bf73baa509de"
server_key_version: "encryptionkey@0"
I20260620 02:21:36.838624 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:36.881932 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:36.887920 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:36.888298 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:36.889564 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:36.891626 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:36.891752 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:36.891919 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:36.892017 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:36.932065 14355 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:45169
I20260620 02:21:36.932149 14481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:45169 every 8 connection(s)
I20260620 02:21:36.933316 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:36.939152 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 14355
I20260620 02:21:36.939383 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:36.940049 10141 external_mini_cluster.cc:1428] Setting key 8cdb78813513c8fdd03216e5b87f3e8f
Jun 20 02:21:36 dist-test-slave-9vg1 krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922096, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:36.958551 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:36.936524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:36543 (local address 127.9.231.126:40015)
0620 02:21:36.937055 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:36.937060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:36.938205 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 02:21:36.940434 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:36.940440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:36.940548 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:36.940684 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:36.942683 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:36.943547 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:36.945473 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:36.945916 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:36.946074 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:36.950418 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:36.950428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:36.950436 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:36.950490 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:36.954445 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:36.955564 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:36.955574 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:36.955579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:36.955766 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:36.956315 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:36.956323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:36.956329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:36.956745 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:36.956919 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:36.957712 (+ 793us) server_negotiation.cc:299] Negotiation successful
0620 02:21:36.958013 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":269,"threads_started":1}
I20260620 02:21:36.959476 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:36.937167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40015 (local address 127.9.231.65:36543)
0620 02:21:36.937948 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:36.937989 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:36.940048 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:36.940952 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:36.940962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:36.941752 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:36.942460 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:36.942509 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:36.943766 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:36.943771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:36.945276 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:36.945293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:36.946200 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:36.946208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:36.946470 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:36.947286 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:36.947313 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:36.950075 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:36.954698 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:36.954708 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:36.954726 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:36.955404 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:36.955936 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:36.955941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:36.955943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:36.956170 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:36.956899 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:36.956906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:36.957268 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 02:21:36.957528 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 02:21:36.957980 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":318,"threads_started":1}
I20260620 02:21:36.961486 14482 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40015
I20260620 02:21:36.961853 14482 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:21:36.962661 14482 heartbeater.cc:511] Master 127.9.231.126:40015 requested a full tablet report, sending...
I20260620 02:21:36.965131 14294 ts_manager.cc:194] Registered new tserver with Master: 97ffde688585457ab4d3726fc239b478 (127.9.231.65:45169)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:21:36.966727 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:36543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:36.996132 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40015
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:37.349220 14488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:37.349536 14488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:37.349691 14488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:37.357409 14488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:37.357537 14488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:37.357657 14488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:37.368075 14488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:37.369454 14488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:37.370769 14488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:37.385502 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:37.385417 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:37.385476 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:37.386806 14488 server_base.cc:1034] running on GCE node
I20260620 02:21:37.387559 14488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:37.388626 14488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:37.389881 14488 hybrid_clock.cc:648] HybridClock initialized: now 1781922097389828 us; error 52 us; skew 500 ppm
I20260620 02:21:37.390245 14488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:37 dist-test-slave-9vg1 krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:37.393748 14488 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:37.395298 14488 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:37.395444 14488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:37.395805 14488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:37.398980 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "16ea7e55cdb4458cb4dc66c510e5cd46"
format_stamp: "Formatted at 2026-06-20 02:21:37 on dist-test-slave-9vg1"
server_key: "10495e2d962645706b847c0514f82f9a"
server_key_iv: "3fdd45fdfafe8b58f00cf389dae6f52f"
server_key_version: "encryptionkey@0"
I20260620 02:21:37.399991 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "16ea7e55cdb4458cb4dc66c510e5cd46"
format_stamp: "Formatted at 2026-06-20 02:21:37 on dist-test-slave-9vg1"
server_key: "10495e2d962645706b847c0514f82f9a"
server_key_iv: "3fdd45fdfafe8b58f00cf389dae6f52f"
server_key_version: "encryptionkey@0"
I20260620 02:21:37.405874 14488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:21:37.410276 14502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:37.412118 14488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 02:21:37.412348 14488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "16ea7e55cdb4458cb4dc66c510e5cd46"
format_stamp: "Formatted at 2026-06-20 02:21:37 on dist-test-slave-9vg1"
server_key: "10495e2d962645706b847c0514f82f9a"
server_key_iv: "3fdd45fdfafe8b58f00cf389dae6f52f"
server_key_version: "encryptionkey@0"
I20260620 02:21:37.412607 14488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:37.429821 14488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:37.434563 14488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:37.434886 14488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:37.436110 14488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:37.437675 14488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:37.437803 14488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:37.437925 14488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:37.437991 14488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:37.474831 14488 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:34723
I20260620 02:21:37.474884 14615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:34723 every 8 connection(s)
I20260620 02:21:37.475837 14488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:37.479096 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 14488
I20260620 02:21:37.479312 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:37.479892 10141 external_mini_cluster.cc:1428] Setting key 3a637407bc0c6f5a41ae562f3ed205b0
Jun 20 02:21:37 dist-test-slave-9vg1 krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:37.501304 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:37.478590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:57733 (local address 127.9.231.126:40015)
0620 02:21:37.479225 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:37.479231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:37.480379 (+ 1148us) server_negotiation.cc:407] Connection header received
0620 02:21:37.482705 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:37.482712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:37.482780 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:37.482873 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:37.485258 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:37.485991 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:37.488659 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:37.489146 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:37.489394 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:37.494132 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:37.494138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:37.494142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:37.494181 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:37.497306 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:37.498556 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:37.498561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:37.498563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:37.498718 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:37.499330 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:37.499339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:37.499341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:37.499624 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:37.499744 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:37.500813 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0620 02:21:37.501004 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:21:37.502782 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:37.479262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40015 (local address 127.9.231.66:57733)
0620 02:21:37.480168 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:37.480222 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:37.482351 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:37.483221 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:37.483238 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:37.484087 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:37.485014 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:37.485050 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:37.486160 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:37.486169 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:37.488443 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:37.488463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:37.489505 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:37.489512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:37.489819 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:37.490842 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:37.490878 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:37.493813 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:37.497532 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:37.497546 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:37.497576 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:37.498387 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:37.498847 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:37.498855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:37.498862 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:37.499193 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:37.499767 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:37.499776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:37.500297 (+ 521us) client_negotiation.cc:769] Sending connection context
0620 02:21:37.500670 (+ 373us) client_negotiation.cc:240] Negotiation successful
0620 02:21:37.501316 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":351,"threads_started":1}
I20260620 02:21:37.504436 14616 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40015
I20260620 02:21:37.504830 14616 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
I20260620 02:21:37.505728 14616 heartbeater.cc:511] Master 127.9.231.126:40015 requested a full tablet report, sending...
I20260620 02:21:37.507647 14294 ts_manager.cc:194] Registered new tserver with Master: 16ea7e55cdb4458cb4dc66c510e5cd46 (127.9.231.66:34723)
I20260620 02:21:37.508549 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:57733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:37.530788 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40015
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:37.879850 14622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:37.880167 14622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:37.880344 14622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:37.887810 14622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:37.887943 14622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:37.888149 14622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:37.898792 14622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:37.900293 14622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:37.901643 14622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:37.915668 14627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:37.915683 14628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:37.915724 14630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:37.916997 14622 server_base.cc:1034] running on GCE node
I20260620 02:21:37.917869 14622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:37.919155 14622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:37.920468 14622 hybrid_clock.cc:648] HybridClock initialized: now 1781922097920396 us; error 60 us; skew 500 ppm
I20260620 02:21:37.920801 14622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:37 dist-test-slave-9vg1 krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:37.924381 14622 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:37.925750 14622 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:37.925849 14622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:37.926236 14622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:37.929505 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "d47ac15d4cc9483997227665e1116e4b"
format_stamp: "Formatted at 2026-06-20 02:21:37 on dist-test-slave-9vg1"
server_key: "16923c34723c6445b7bfce8d0428bbb7"
server_key_iv: "9bcab8c1b6977d7a9cc63c14cb977e56"
server_key_version: "encryptionkey@0"
I20260620 02:21:37.930402 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "d47ac15d4cc9483997227665e1116e4b"
format_stamp: "Formatted at 2026-06-20 02:21:37 on dist-test-slave-9vg1"
server_key: "16923c34723c6445b7bfce8d0428bbb7"
server_key_iv: "9bcab8c1b6977d7a9cc63c14cb977e56"
server_key_version: "encryptionkey@0"
I20260620 02:21:37.936486 14622 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:21:37.940879 14636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:37.942814 14622 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260620 02:21:37.943166 14622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "d47ac15d4cc9483997227665e1116e4b"
format_stamp: "Formatted at 2026-06-20 02:21:37 on dist-test-slave-9vg1"
server_key: "16923c34723c6445b7bfce8d0428bbb7"
server_key_iv: "9bcab8c1b6977d7a9cc63c14cb977e56"
server_key_version: "encryptionkey@0"
I20260620 02:21:37.943423 14622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:37.966403 14622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:37.970207 14482 heartbeater.cc:503] Master 127.9.231.126:40015 was elected leader, sending a full tablet report...
I20260620 02:21:37.971632 14622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:37.971959 14622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:37.973119 14622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:37.974596 14622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:37.974709 14622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:37.974852 14622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:37.974916 14622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:38.011636 14622 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:39223
I20260620 02:21:38.011729 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:39223 every 8 connection(s)
I20260620 02:21:38.012526 14622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:38.013432 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 14622
I20260620 02:21:38.013640 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:38.014171 10141 external_mini_cluster.cc:1428] Setting key 3cb8161e58164e6f9d95e4a72e02919d
Jun 20 02:21:38 dist-test-slave-9vg1 krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:38.033730 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.015412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:43021 (local address 127.9.231.126:40015)
0620 02:21:38.015907 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.015913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.017010 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 02:21:38.018875 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.018879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.018959 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.019053 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:38.020660 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.021424 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.022992 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.023510 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.023610 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.027428 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:38.027434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:38.027437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:38.027473 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:38.030689 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.031571 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.031575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.031577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.031706 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.032095 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.032098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.032100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.032372 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:38.032482 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.033241 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.033430 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":226,"threads_started":1}
I20260620 02:21:38.034790 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.016012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40015 (local address 127.9.231.67:43021)
0620 02:21:38.016861 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:38.016907 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:38.018621 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:38.019385 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:38.019394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:38.019916 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:38.020493 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.020513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.021568 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.021572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.022859 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.022872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.023721 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.023726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.023927 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.024629 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:38.024678 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:38.027182 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:38.030832 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:38.030840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:38.030857 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:38.031436 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:38.031809 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:38.031812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:38.031815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:38.031999 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:38.032486 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:38.032492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:38.032842 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 02:21:38.033077 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 02:21:38.033515 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":370,"threads_started":1}
I20260620 02:21:38.036674 14750 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40015
I20260620 02:21:38.037016 14750 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:38.037838 14750 heartbeater.cc:511] Master 127.9.231.126:40015 requested a full tablet report, sending...
I20260620 02:21:38.039453 14294 ts_manager.cc:194] Registered new tserver with Master: d47ac15d4cc9483997227665e1116e4b (127.9.231.67:39223)
I20260620 02:21:38.040297 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:43021
I20260620 02:21:38.043396 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:21:38.061889 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.048551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.9.231.126:40015)
0620 02:21:38.048760 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.048765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.049435 (+ 670us) server_negotiation.cc:407] Connection header received
0620 02:21:38.049588 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.049592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.049664 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.049773 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:38.051421 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.052150 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.053428 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.053823 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.053921 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.056374 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:38.056379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:38.056382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:38.056417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:38.059129 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.059840 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.059844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.059846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.059975 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.060408 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.060413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.060415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.060824 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:38.061030 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.061419 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.061643 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:21:38.066684 14294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:38.074168 14294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:38.111280 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.088110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45169 (local address 127.0.0.1:51408)
0620 02:21:38.099202 (+ 11092us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:38.099273 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:38.100604 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:38.102123 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:38.102149 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:38.102199 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:38.102758 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.102773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.105366 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.105371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.108014 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.108040 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.110094 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.110100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.110330 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.110386 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:21:38.110556 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 02:21:38.110819 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10652,"thread_start_us":11324,"threads_started":1}
I20260620 02:21:38.111666 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.088256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.9.231.65:45169)
0620 02:21:38.095662 (+ 7406us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.095679 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.099457 (+ 3778us) server_negotiation.cc:407] Connection header received
0620 02:21:38.100960 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.100974 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.101354 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.101598 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:38.102972 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.105145 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.108297 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.109863 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.110000 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.110259 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.111121 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.111324 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6667,"thread_start_us":6497,"threads_started":1}
I20260620 02:21:38.113046 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.089253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.9.231.66:34723)
0620 02:21:38.103956 (+ 14703us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.103969 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.104016 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:21:38.104171 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.104184 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.104519 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.104842 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:38.105581 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.107252 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.109984 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.111487 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.111658 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.111888 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.112488 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.112678 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14518,"thread_start_us":14354,"threads_started":1}
I20260620 02:21:38.112907 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.088328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:34723 (local address 127.0.0.1:35994)
0620 02:21:38.102873 (+ 14545us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:38.102895 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:38.103005 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:38.104794 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:38.104800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:38.104848 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:38.105352 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.105365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.107451 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.107457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.109735 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.109752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.112055 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.112064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.112258 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.112293 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:21:38.112399 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:21:38.112538 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14383,"thread_start_us":14174,"threads_started":1}
I20260620 02:21:38.116147 14417 tablet_service.cc:1467] Processing CreateTablet for tablet a77112a92cf94a479ef124753b46bff8 (DEFAULT_TABLE table=test-table [id=4eacb14c7a3a49ae8b05a86c61a98c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:38.117843 14417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77112a92cf94a479ef124753b46bff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:38.118963 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.088764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:39223 (local address 127.0.0.1:58408)
0620 02:21:38.108287 (+ 19523us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:38.108317 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:38.108503 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:38.110353 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:38.110358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:38.110387 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:38.110826 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.110837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.112886 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.112897 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.116093 (+ 3196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.116110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.118057 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.118065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.118301 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.118337 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:21:38.118443 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:21:38.118597 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19362,"thread_start_us":19153,"threads_started":1}
I20260620 02:21:38.119323 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.089008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.9.231.67:39223)
0620 02:21:38.107864 (+ 18856us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.107878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.108445 (+ 567us) server_negotiation.cc:407] Connection header received
0620 02:21:38.108782 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.108793 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.109161 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.109364 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:38.111102 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.112685 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.116274 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.117761 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.117925 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.118170 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.118764 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.118926 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18598,"thread_start_us":20652,"threads_started":1}
I20260620 02:21:38.118968 14550 tablet_service.cc:1467] Processing CreateTablet for tablet a77112a92cf94a479ef124753b46bff8 (DEFAULT_TABLE table=test-table [id=4eacb14c7a3a49ae8b05a86c61a98c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:38.120600 14550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77112a92cf94a479ef124753b46bff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:38.127089 14685 tablet_service.cc:1467] Processing CreateTablet for tablet a77112a92cf94a479ef124753b46bff8 (DEFAULT_TABLE table=test-table [id=4eacb14c7a3a49ae8b05a86c61a98c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:38.128458 14685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77112a92cf94a479ef124753b46bff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:38.135493 14769 tablet_bootstrap.cc:492] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Bootstrap starting.
I20260620 02:21:38.138012 14770 tablet_bootstrap.cc:492] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Bootstrap starting.
I20260620 02:21:38.139531 14769 tablet_bootstrap.cc:654] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:38.140941 14769 log.cc:824] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:38.141273 14770 tablet_bootstrap.cc:654] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:38.142746 14770 log.cc:824] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:38.144860 14772 tablet_bootstrap.cc:492] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Bootstrap starting.
I20260620 02:21:38.144945 14769 tablet_bootstrap.cc:492] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: No bootstrap required, opened a new log
I20260620 02:21:38.145318 14769 ts_tablet_manager.cc:1397] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:21:38.148397 14770 tablet_bootstrap.cc:492] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: No bootstrap required, opened a new log
I20260620 02:21:38.148721 14770 ts_tablet_manager.cc:1397] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 02:21:38.149256 14772 tablet_bootstrap.cc:654] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:38.150345 14772 log.cc:824] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:38.153138 14772 tablet_bootstrap.cc:492] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: No bootstrap required, opened a new log
I20260620 02:21:38.153569 14772 ts_tablet_manager.cc:1397] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260620 02:21:38.156656 14769 raft_consensus.cc:348] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.157150 14769 raft_consensus.cc:374] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:38.157294 14769 raft_consensus.cc:729] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ffde688585457ab4d3726fc239b478, State: Initialized, Role: FOLLOWER
I20260620 02:21:38.157974 14769 consensus_queue.cc:260] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.159866 14770 raft_consensus.cc:348] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.160315 14770 raft_consensus.cc:374] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:38.160578 14770 raft_consensus.cc:729] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ea7e55cdb4458cb4dc66c510e5cd46, State: Initialized, Role: FOLLOWER
I20260620 02:21:38.161387 14770 consensus_queue.cc:260] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.161650 14769 ts_tablet_manager.cc:1428] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260620 02:21:38.162650 14616 heartbeater.cc:503] Master 127.9.231.126:40015 was elected leader, sending a full tablet report...
I20260620 02:21:38.162866 14772 raft_consensus.cc:348] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.163422 14772 raft_consensus.cc:374] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:38.163450 14770 ts_tablet_manager.cc:1428] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 02:21:38.163582 14772 raft_consensus.cc:729] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47ac15d4cc9483997227665e1116e4b, State: Initialized, Role: FOLLOWER
I20260620 02:21:38.164354 14772 consensus_queue.cc:260] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.166193 14750 heartbeater.cc:503] Master 127.9.231.126:40015 was elected leader, sending a full tablet report...
I20260620 02:21:38.166501 14772 ts_tablet_manager.cc:1428] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260620 02:21:38.186026 14483 tablet.cc:2369] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:21:38.227963 14617 tablet.cc:2369] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:21:38.264384 14751 tablet.cc:2369] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:38.433459 14776 raft_consensus.cc:482] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:38.433794 14776 raft_consensus.cc:504] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.435992 14776 leader_election.cc:288] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ffde688585457ab4d3726fc239b478 (127.9.231.65:45169), d47ac15d4cc9483997227665e1116e4b (127.9.231.67:39223)
I20260620 02:21:38.443635 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.436446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:53029 (local address 127.9.231.65:45169)
0620 02:21:38.436744 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.436751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.437193 (+ 442us) server_negotiation.cc:407] Connection header received
0620 02:21:38.437445 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.437451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.437545 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.437687 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:38.438363 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.439400 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.441490 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.442492 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.442635 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.442699 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.443164 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.443334 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 02:21:38.443854 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.436216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45169 (local address 127.9.231.66:53029)
0620 02:21:38.437014 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:38.437040 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:38.437261 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:38.437746 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:38.437751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:38.437789 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:38.438185 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.438201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.439534 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.439539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.441286 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.441302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.442719 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.442727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.442932 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.442968 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:21:38.443087 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 02:21:38.443243 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":443,"threads_started":1}
I20260620 02:21:38.444221 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.436365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:39223 (local address 127.9.231.66:47859)
0620 02:21:38.437613 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:38.437637 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:38.437771 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:38.438234 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:38.438237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:38.438293 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:38.438687 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.438701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.439999 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.440004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.441690 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:38.441707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.443321 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.443327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:38.443561 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.443598 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:21:38.443713 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:21:38.443872 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"thread_start_us":857,"threads_started":1}
I20260620 02:21:38.444273 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.436774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:47859 (local address 127.9.231.67:39223)
0620 02:21:38.436977 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.436982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.437737 (+ 755us) server_negotiation.cc:407] Connection header received
0620 02:21:38.437921 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.437929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.438008 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.438120 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:38.438899 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.439812 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.441884 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.442951 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.443159 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.443226 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.443749 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.443909 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:21:38.445262 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77112a92cf94a479ef124753b46bff8" candidate_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ffde688585457ab4d3726fc239b478" is_pre_election: true
I20260620 02:21:38.445456 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77112a92cf94a479ef124753b46bff8" candidate_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47ac15d4cc9483997227665e1116e4b" is_pre_election: true
I20260620 02:21:38.445869 14437 raft_consensus.cc:2454] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ea7e55cdb4458cb4dc66c510e5cd46 in term 0.
I20260620 02:21:38.446004 14705 raft_consensus.cc:2454] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ea7e55cdb4458cb4dc66c510e5cd46 in term 0.
I20260620 02:21:38.446655 14503 leader_election.cc:302] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ea7e55cdb4458cb4dc66c510e5cd46, 97ffde688585457ab4d3726fc239b478; no voters:
I20260620 02:21:38.447122 14776 raft_consensus.cc:2781] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:38.447293 14776 raft_consensus.cc:482] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:38.447414 14776 raft_consensus.cc:3037] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:38.449836 14776 raft_consensus.cc:504] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.450601 14776 leader_election.cc:288] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [CANDIDATE]: Term 1 election: Requested vote from peers 97ffde688585457ab4d3726fc239b478 (127.9.231.65:45169), d47ac15d4cc9483997227665e1116e4b (127.9.231.67:39223)
I20260620 02:21:38.451248 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77112a92cf94a479ef124753b46bff8" candidate_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47ac15d4cc9483997227665e1116e4b"
I20260620 02:21:38.451220 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77112a92cf94a479ef124753b46bff8" candidate_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ffde688585457ab4d3726fc239b478"
I20260620 02:21:38.451493 14705 raft_consensus.cc:3037] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:38.451516 14437 raft_consensus.cc:3037] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:38.453840 14705 raft_consensus.cc:2454] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ea7e55cdb4458cb4dc66c510e5cd46 in term 1.
I20260620 02:21:38.454200 14437 raft_consensus.cc:2454] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ea7e55cdb4458cb4dc66c510e5cd46 in term 1.
I20260620 02:21:38.454422 14504 leader_election.cc:302] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ea7e55cdb4458cb4dc66c510e5cd46, d47ac15d4cc9483997227665e1116e4b; no voters:
I20260620 02:21:38.454833 14776 raft_consensus.cc:2781] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:38.455353 14776 raft_consensus.cc:686] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 1 LEADER]: Becoming Leader. State: Replica: 16ea7e55cdb4458cb4dc66c510e5cd46, State: Running, Role: LEADER
I20260620 02:21:38.455878 14776 consensus_queue.cc:237] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } }
I20260620 02:21:38.461382 14291 catalog_manager.cc:5495] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16ea7e55cdb4458cb4dc66c510e5cd46 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ea7e55cdb4458cb4dc66c510e5cd46" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 } health_report { overall_health: UNKNOWN } } }
I20260620 02:21:38.489301 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.482339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.9.231.66:34723)
0620 02:21:38.482642 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.482651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.482720 (+ 69us) server_negotiation.cc:407] Connection header received
0620 02:21:38.482981 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.482987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.483208 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.483368 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:38.483854 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.484665 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.486640 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.487281 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.487398 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.488033 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:38.488591 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:38.488711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.488860 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.488934 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 02:21:38.499037 14705 raft_consensus.cc:1261] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 1 FOLLOWER]: Refusing update from remote peer 16ea7e55cdb4458cb4dc66c510e5cd46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:38.499037 14437 raft_consensus.cc:1261] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 1 FOLLOWER]: Refusing update from remote peer 16ea7e55cdb4458cb4dc66c510e5cd46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:38.500438 14780 consensus_queue.cc:1035] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ffde688585457ab4d3726fc239b478" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:38.500916 14776 consensus_queue.cc:1035] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d47ac15d4cc9483997227665e1116e4b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:38.514693 14783 mvcc.cc:205] Tried to move back new op lower bound from 7298752915438587904 to 7298752915279507456. Current Snapshot: MvccSnapshot[applied={T|T < 7298752915438587904}]
I20260620 02:21:38.534197 14292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:38.534590 14292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:38.538986 14292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14532d8c158d42eba1339ac400099148: Sending DeleteTablet for 3 replicas of tablet a77112a92cf94a479ef124753b46bff8
I20260620 02:21:38.540459 14417 tablet_service.cc:1514] Processing DeleteTablet for tablet a77112a92cf94a479ef124753b46bff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:38 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:51408
I20260620 02:21:38.540467 14550 tablet_service.cc:1514] Processing DeleteTablet for tablet a77112a92cf94a479ef124753b46bff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:38 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:35994
I20260620 02:21:38.540932 14685 tablet_service.cc:1514] Processing DeleteTablet for tablet a77112a92cf94a479ef124753b46bff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:38 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:58408
I20260620 02:21:38.543432 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 14355
I20260620 02:21:38.554289 14794 tablet_replica.cc:330] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: stopping tablet replica
I20260620 02:21:38.555027 14793 tablet_replica.cc:330] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: stopping tablet replica
I20260620 02:21:38.555944 14793 raft_consensus.cc:2229] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:38.556618 14794 raft_consensus.cc:2229] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:38.557422 14793 raft_consensus.cc:2258] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:38.558363 14794 raft_consensus.cc:2258] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:38.561245 14793 ts_tablet_manager.cc:1905] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:38.564014 14794 ts_tablet_manager.cc:1905] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:21:38 dist-test-slave-9vg1 krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:38.570291 14796 tablet_replica.cc:330] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: stopping tablet replica
I20260620 02:21:38.570807 14796 raft_consensus.cc:2229] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:38.571432 14796 raft_consensus.cc:2258] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:38.573220 14793 ts_tablet_manager.cc:1918] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:38.573537 14793 log.cc:1196] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/a77112a92cf94a479ef124753b46bff8
I20260620 02:21:38.574239 14793 ts_tablet_manager.cc:1939] T a77112a92cf94a479ef124753b46bff8 P 97ffde688585457ab4d3726fc239b478: Deleting consensus metadata
I20260620 02:21:38.576946 14278 catalog_manager.cc:4841] TS 97ffde688585457ab4d3726fc239b478 (127.9.231.65:45169): tablet a77112a92cf94a479ef124753b46bff8 (table test-table [id=4eacb14c7a3a49ae8b05a86c61a98c4a]) successfully deleted
I20260620 02:21:38.578455 14794 ts_tablet_manager.cc:1918] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:38.578660 14794 log.cc:1196] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/a77112a92cf94a479ef124753b46bff8
I20260620 02:21:38.579216 14796 ts_tablet_manager.cc:1905] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:38.579357 14794 ts_tablet_manager.cc:1939] T a77112a92cf94a479ef124753b46bff8 P 16ea7e55cdb4458cb4dc66c510e5cd46: Deleting consensus metadata
I20260620 02:21:38.581378 14281 catalog_manager.cc:4841] TS 16ea7e55cdb4458cb4dc66c510e5cd46 (127.9.231.66:34723): tablet a77112a92cf94a479ef124753b46bff8 (table test-table [id=4eacb14c7a3a49ae8b05a86c61a98c4a]) successfully deleted
I20260620 02:21:38.581673 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.545124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.9.231.65:45169)
0620 02:21:38.545353 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.545362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.559037 (+ 13675us) server_negotiation.cc:407] Connection header received
0620 02:21:38.559584 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.559590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.559711 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.559796 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:38.561639 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.562716 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.564575 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.565247 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.565356 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.573126 (+ 7770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:38.573148 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:38.573168 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:38.573239 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:38.577660 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.578717 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.578730 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.578733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.578928 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.579516 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.579524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.579530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.580016 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:38.580349 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.580980 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.581268 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:21:38.583498 14477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:38.586841 14796 ts_tablet_manager.cc:1918] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:38.587078 14796 log.cc:1196] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/a77112a92cf94a479ef124753b46bff8
I20260620 02:21:38.587658 14796 ts_tablet_manager.cc:1939] T a77112a92cf94a479ef124753b46bff8 P d47ac15d4cc9483997227665e1116e4b: Deleting consensus metadata
I20260620 02:21:38.589798 14279 catalog_manager.cc:4841] TS d47ac15d4cc9483997227665e1116e4b (127.9.231.67:39223): tablet a77112a92cf94a479ef124753b46bff8 (table test-table [id=4eacb14c7a3a49ae8b05a86c61a98c4a]) successfully deleted
I20260620 02:21:38.678558 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 14355
I20260620 02:21:38.697517 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 14488
Jun 20 02:21:38 dist-test-slave-9vg1 krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:21:38.715142 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.698375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.9.231.66:34723)
0620 02:21:38.698597 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.698603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.698700 (+ 97us) server_negotiation.cc:407] Connection header received
0620 02:21:38.698922 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.698928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.699019 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.699185 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:38.700434 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.701246 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.702525 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.703014 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.703364 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.707967 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:38.707985 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:38.708003 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:38.708059 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:38.711843 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.712698 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.712713 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.712716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.712910 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.713364 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.713370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.713373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.713756 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:38.714150 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.714398 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.714645 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:21:38.715917 14611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:38.833266 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 14488
I20260620 02:21:38.853653 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 14622
Jun 20 02:21:38 dist-test-slave-9vg1 krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:21:38.875044 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:38.854532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.9.231.67:39223)
0620 02:21:38.854716 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:38.854720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:38.854758 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:21:38.854971 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:38.854976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:38.855216 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:38.855315 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:38.856935 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.857719 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.860471 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:38.861290 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:38.861455 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:38.868608 (+ 7153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:38.868661 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:38.868679 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:38.868738 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:38.871881 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.872745 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.872754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.872756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.872918 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:38.873306 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:38.873309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:38.873312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:38.873733 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:38.873980 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:38.874425 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 02:21:38.874697 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:21:38.875959 14745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:38.975453 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 14622
I20260620 02:21:39.002481 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 14263
I20260620 02:21:39.003752 14324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:39.116981 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 14263
2026-06-20T02:21:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4718 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T02:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:39Z Disabled control of system clock
I20260620 02:21:39.194356 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33325
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:21:39.560088 14805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:39.560451 14805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:39.560606 14805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:39.568440 14805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:39.568572 14805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:39.568655 14805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:39.568739 14805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:39.568820 14805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:39.579008 14805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:39.580580 14805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:39.582597 14805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:39.596863 14813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:39.596931 14810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:39.596894 14811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:39.597692 14805 server_base.cc:1034] running on GCE node
I20260620 02:21:39.598670 14805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:39.600412 14805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:39.601691 14805 hybrid_clock.cc:648] HybridClock initialized: now 1781922099601648 us; error 36 us; skew 500 ppm
I20260620 02:21:39.602147 14805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:39.609913 14805 webserver.cc:460] Webserver started at http://127.9.231.126:40171/ using document root <none> and password file <none>
I20260620 02:21:39.610803 14805 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:39.610910 14805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:39.611338 14805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:39.614579 14805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "953bd671bfbb4bc9a66fec7dd518f61e"
format_stamp: "Formatted at 2026-06-20 02:21:39 on dist-test-slave-9vg1"
server_key: "e87ff56d1c790afca0eee7a781dc212a"
server_key_iv: "67cd9a7f953de96c7714b8fb26e751c0"
server_key_version: "encryptionkey@0"
I20260620 02:21:39.615610 14805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "953bd671bfbb4bc9a66fec7dd518f61e"
format_stamp: "Formatted at 2026-06-20 02:21:39 on dist-test-slave-9vg1"
server_key: "e87ff56d1c790afca0eee7a781dc212a"
server_key_iv: "67cd9a7f953de96c7714b8fb26e751c0"
server_key_version: "encryptionkey@0"
I20260620 02:21:39.621456 14805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:21:39.625414 14819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:39.627311 14805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:21:39.627619 14805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "953bd671bfbb4bc9a66fec7dd518f61e"
format_stamp: "Formatted at 2026-06-20 02:21:39 on dist-test-slave-9vg1"
server_key: "e87ff56d1c790afca0eee7a781dc212a"
server_key_iv: "67cd9a7f953de96c7714b8fb26e751c0"
server_key_version: "encryptionkey@0"
I20260620 02:21:39.627913 14805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:39.635154 14805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:39.636142 14805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:39.636511 14805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:39.658473 14805 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33325
I20260620 02:21:39.658490 14870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33325 every 8 connection(s)
I20260620 02:21:39.660127 14805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:39.664474 14871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:39.668082 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 14805
I20260620 02:21:39.668304 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:39.669126 10141 external_mini_cluster.cc:1428] Setting key c255df47365320d68ac4cd8dabf60b00
I20260620 02:21:39.678318 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:39.671578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.9.231.126:33325)
0620 02:21:39.672647 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:39.672658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:39.672702 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:21:39.674330 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:39.674354 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:39.674827 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:39.675334 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:39.676539 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:39.676546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:39.676559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:39.676580 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:39.676634 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:39.676650 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:39.676894 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:39.677037 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 02:21:39.677313 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":387,"threads_started":1}
I20260620 02:21:39.678339 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e: Bootstrap starting.
I20260620 02:21:39.682109 14871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:39.683398 14871 log.cc:824] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:39.687175 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e: No bootstrap required, opened a new log
I20260620 02:21:39.695915 14871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33325 } }
I20260620 02:21:39.696381 14871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:39.696496 14871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953bd671bfbb4bc9a66fec7dd518f61e, State: Initialized, Role: FOLLOWER
I20260620 02:21:39.697139 14871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33325 } }
I20260620 02:21:39.697408 14871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:39.697576 14871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:39.697717 14871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:39.700515 14871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33325 } }
I20260620 02:21:39.701049 14871 leader_election.cc:302] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 953bd671bfbb4bc9a66fec7dd518f61e; no voters:
I20260620 02:21:39.701650 14871 leader_election.cc:288] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:39.701815 14876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:39.702553 14876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [term 1 LEADER]: Becoming Leader. State: Replica: 953bd671bfbb4bc9a66fec7dd518f61e, State: Running, Role: LEADER
I20260620 02:21:39.703236 14876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33325 } }
I20260620 02:21:39.703925 14871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:39.706207 14878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 953bd671bfbb4bc9a66fec7dd518f61e. Latest consensus state: current_term: 1 leader_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33325 } } }
I20260620 02:21:39.706621 14878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:39.706341 14877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953bd671bfbb4bc9a66fec7dd518f61e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33325 } } }
I20260620 02:21:39.707046 14877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:39.709226 14884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:39.715379 14884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:39.725194 14884 catalog_manager.cc:1347] Generated new cluster ID: 94d0b211e91743819ef4801f8622754c
I20260620 02:21:39.725346 14884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:39.745281 14884 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:39.745886 14884 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:39.768731 14884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e: Generated new TSK 0
I20260620 02:21:39.769984 14884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:21:39.783828 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33325
--builtin_ntp_servers=127.9.231.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:21:40.145474 14895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:40.145861 14895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:40.146019 14895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:40.153868 14895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:40.154063 14895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:40.154250 14895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:40.164294 14895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:40.165764 14895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:40.167805 14895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:40.182806 14900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:40.182833 14903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:40.182783 14901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:40.184466 14895 server_base.cc:1034] running on GCE node
I20260620 02:21:40.185217 14895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:40.186229 14895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:40.187469 14895 hybrid_clock.cc:648] HybridClock initialized: now 1781922100187416 us; error 38 us; skew 500 ppm
I20260620 02:21:40.187811 14895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:40.190433 14895 webserver.cc:460] Webserver started at http://127.9.231.65:35837/ using document root <none> and password file <none>
I20260620 02:21:40.191287 14895 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:40.191431 14895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:40.191777 14895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:40.194824 14895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "9cd99ed07865453bbc4a14606a4f95d0"
format_stamp: "Formatted at 2026-06-20 02:21:40 on dist-test-slave-9vg1"
server_key: "95925517307752055b0224813757d6b1"
server_key_iv: "1c2e14dc4bbf35d7a02cd84702171dcc"
server_key_version: "encryptionkey@0"
I20260620 02:21:40.195773 14895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd99ed07865453bbc4a14606a4f95d0"
format_stamp: "Formatted at 2026-06-20 02:21:40 on dist-test-slave-9vg1"
server_key: "95925517307752055b0224813757d6b1"
server_key_iv: "1c2e14dc4bbf35d7a02cd84702171dcc"
server_key_version: "encryptionkey@0"
I20260620 02:21:40.201530 14895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:21:40.205698 14909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:40.207531 14895 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:21:40.207811 14895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "9cd99ed07865453bbc4a14606a4f95d0"
format_stamp: "Formatted at 2026-06-20 02:21:40 on dist-test-slave-9vg1"
server_key: "95925517307752055b0224813757d6b1"
server_key_iv: "1c2e14dc4bbf35d7a02cd84702171dcc"
server_key_version: "encryptionkey@0"
I20260620 02:21:40.208091 14895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:40.214423 14895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:40.215466 14895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:40.215767 14895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:40.216897 14895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:40.218492 14895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:40.218617 14895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:40.218758 14895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:40.218860 14895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:40.257748 14895 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:40543
I20260620 02:21:40.257858 15022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:40543 every 8 connection(s)
I20260620 02:21:40.259328 14895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:40.268719 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 14895
I20260620 02:21:40.268958 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:40.269001 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:40.262350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:34071 (local address 127.9.231.126:33325)
0620 02:21:40.262848 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:40.262853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:40.264299 (+ 1446us) server_negotiation.cc:407] Connection header received
0620 02:21:40.266116 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:40.266121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:40.266184 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:40.266288 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:40.267905 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:40.267911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:40.267914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:40.267934 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:40.267988 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:40.268002 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:40.268267 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:40.268503 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 02:21:40.268589 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":273,"threads_started":1}
I20260620 02:21:40.269716 10141 external_mini_cluster.cc:1428] Setting key bfb87f3d1a5d782f71280eab1d7dfc9b
I20260620 02:21:40.270202 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:40.263022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33325 (local address 127.9.231.65:34071)
0620 02:21:40.264079 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:40.264125 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:40.265772 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:40.266568 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:40.266578 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:40.266693 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:40.267542 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:40.267545 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:40.267588 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:40.267688 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:40.268247 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:40.268257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:40.268264 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:21:40.268530 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 02:21:40.268896 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":306,"threads_started":1}
I20260620 02:21:40.272111 15023 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33325
I20260620 02:21:40.272488 15023 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:40.273074 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33325
--builtin_ntp_servers=127.9.231.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:21:40.273190 15023 heartbeater.cc:511] Master 127.9.231.126:33325 requested a full tablet report, sending...
I20260620 02:21:40.275648 14836 ts_manager.cc:194] Registered new tserver with Master: 9cd99ed07865453bbc4a14606a4f95d0 (127.9.231.65:40543)
W20260620 02:21:40.632087 15027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:40.632546 15027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:40.632722 15027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:40.640749 15027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:40.640923 15027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:40.641106 15027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:40.651345 15027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:40.652859 15027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:40.655009 15027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:40.671749 15035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:40.671725 15033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:40.672659 15032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:40.673220 15027 server_base.cc:1034] running on GCE node
I20260620 02:21:40.673995 15027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:40.675364 15027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:40.676656 15027 hybrid_clock.cc:648] HybridClock initialized: now 1781922100676572 us; error 84 us; skew 500 ppm
I20260620 02:21:40.677059 15027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:40.680162 15027 webserver.cc:460] Webserver started at http://127.9.231.66:34211/ using document root <none> and password file <none>
I20260620 02:21:40.681140 15027 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:40.681282 15027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:40.681675 15027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:40.684980 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "2571c42a24ce4bdcbc66fca9749e1a7f"
format_stamp: "Formatted at 2026-06-20 02:21:40 on dist-test-slave-9vg1"
server_key: "1f7e2740d92217bdd57a79109b36b569"
server_key_iv: "99f6415250cb2dcd3bacb2f46467677e"
server_key_version: "encryptionkey@0"
I20260620 02:21:40.685894 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "2571c42a24ce4bdcbc66fca9749e1a7f"
format_stamp: "Formatted at 2026-06-20 02:21:40 on dist-test-slave-9vg1"
server_key: "1f7e2740d92217bdd57a79109b36b569"
server_key_iv: "99f6415250cb2dcd3bacb2f46467677e"
server_key_version: "encryptionkey@0"
I20260620 02:21:40.691892 15027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:21:40.696050 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:40.697623 15027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:40.697841 15027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "2571c42a24ce4bdcbc66fca9749e1a7f"
format_stamp: "Formatted at 2026-06-20 02:21:40 on dist-test-slave-9vg1"
server_key: "1f7e2740d92217bdd57a79109b36b569"
server_key_iv: "99f6415250cb2dcd3bacb2f46467677e"
server_key_version: "encryptionkey@0"
I20260620 02:21:40.698115 15027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:40.704761 15027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:40.705806 15027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:40.706108 15027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:40.707451 15027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:40.709161 15027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:40.709306 15027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:40.709450 15027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:40.709537 15027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:40.745841 15027 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:45065
I20260620 02:21:40.746100 15154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:45065 every 8 connection(s)
I20260620 02:21:40.747375 15027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:40.756409 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15027
I20260620 02:21:40.756649 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:40.756824 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:40.750510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:44623 (local address 127.9.231.126:33325)
0620 02:21:40.750748 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:40.750754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:40.752233 (+ 1479us) server_negotiation.cc:407] Connection header received
0620 02:21:40.754119 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:40.754125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:40.754203 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:40.754338 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:40.755890 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:40.755895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:40.755899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:40.755921 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:40.755960 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:40.755975 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:40.756191 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:40.756395 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 02:21:40.756458 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 02:21:40.757206 10141 external_mini_cluster.cc:1428] Setting key 35540d6af3083d97ff50533ab11c9f43
I20260620 02:21:40.758144 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:40.750961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33325 (local address 127.9.231.66:44623)
0620 02:21:40.752042 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:40.752089 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:40.753785 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:40.754582 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:40.754591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:40.754705 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:40.755518 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:40.755521 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:40.755566 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:40.755666 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:40.756125 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:40.756139 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:40.756150 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:21:40.756494 (+ 344us) client_negotiation.cc:240] Negotiation successful
0620 02:21:40.756849 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":308,"threads_started":1}
I20260620 02:21:40.759904 15155 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33325
I20260620 02:21:40.760286 15155 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:40.760656 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33325
--builtin_ntp_servers=127.9.231.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:21:40.761039 15155 heartbeater.cc:511] Master 127.9.231.126:33325 requested a full tablet report, sending...
I20260620 02:21:40.762848 14836 ts_manager.cc:194] Registered new tserver with Master: 2571c42a24ce4bdcbc66fca9749e1a7f (127.9.231.66:45065)
W20260620 02:21:41.136855 15158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:41.137243 15158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:41.137421 15158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:41.145237 15158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:41.145407 15158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:41.145593 15158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:41.155759 15158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:41.157488 15158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:41.159610 15158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:41.174993 15163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:41.174954 15166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:41.174917 15164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:41.176103 15158 server_base.cc:1034] running on GCE node
I20260620 02:21:41.176887 15158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:41.178138 15158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:41.179397 15158 hybrid_clock.cc:648] HybridClock initialized: now 1781922101179335 us; error 75 us; skew 500 ppm
I20260620 02:21:41.179776 15158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:41.182740 15158 webserver.cc:460] Webserver started at http://127.9.231.67:39319/ using document root <none> and password file <none>
I20260620 02:21:41.183830 15158 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:41.183991 15158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:41.184486 15158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:41.188091 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "d665f99736c1400797f6cd5d0e1377a3"
format_stamp: "Formatted at 2026-06-20 02:21:41 on dist-test-slave-9vg1"
server_key: "577b3ecff99b5d9c2326276ac728e204"
server_key_iv: "a3fce351cc81376ba919653353a81c68"
server_key_version: "encryptionkey@0"
I20260620 02:21:41.188928 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "d665f99736c1400797f6cd5d0e1377a3"
format_stamp: "Formatted at 2026-06-20 02:21:41 on dist-test-slave-9vg1"
server_key: "577b3ecff99b5d9c2326276ac728e204"
server_key_iv: "a3fce351cc81376ba919653353a81c68"
server_key_version: "encryptionkey@0"
I20260620 02:21:41.194839 15158 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:21:41.199137 15172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:41.200778 15158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:41.200981 15158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "d665f99736c1400797f6cd5d0e1377a3"
format_stamp: "Formatted at 2026-06-20 02:21:41 on dist-test-slave-9vg1"
server_key: "577b3ecff99b5d9c2326276ac728e204"
server_key_iv: "a3fce351cc81376ba919653353a81c68"
server_key_version: "encryptionkey@0"
I20260620 02:21:41.201277 15158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:41.208321 15158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:41.209414 15158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:41.209731 15158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:41.210788 15158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:41.212534 15158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:41.212628 15158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:41.212769 15158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:41.212831 15158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:41.250511 15158 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:38213
I20260620 02:21:41.250560 15286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:38213 every 8 connection(s)
I20260620 02:21:41.252048 15158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:41.256359 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15158
I20260620 02:21:41.256562 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:41.257143 10141 external_mini_cluster.cc:1428] Setting key 7d5114e5d3b177b6090c0d40ed02c82e
I20260620 02:21:41.263396 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.255729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:52223 (local address 127.9.231.126:33325)
0620 02:21:41.255958 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.255967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.258094 (+ 2127us) server_negotiation.cc:407] Connection header received
0620 02:21:41.260622 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.260628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.260707 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.260851 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.262491 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.262498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.262504 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.262529 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.262577 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.262594 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.262737 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.262970 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.263043 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:21:41.264401 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.256245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33325 (local address 127.9.231.67:52223)
0620 02:21:41.257873 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.257931 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:41.260310 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:41.261129 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:41.261141 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:41.261291 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:41.262100 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:41.262102 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:41.262146 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:41.262248 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:41.262716 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:41.262723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:41.262730 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:21:41.263009 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 02:21:41.263311 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":395,"threads_started":1}
I20260620 02:21:41.266258 15287 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33325
I20260620 02:21:41.266606 15287 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:41.267532 15287 heartbeater.cc:511] Master 127.9.231.126:33325 requested a full tablet report, sending...
I20260620 02:21:41.269294 14836 ts_manager.cc:194] Registered new tserver with Master: d665f99736c1400797f6cd5d0e1377a3 (127.9.231.67:38213)
I20260620 02:21:41.274701 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:21:41.279304 15023 heartbeater.cc:503] Master 127.9.231.126:33325 was elected leader, sending a full tablet report...
I20260620 02:21:41.283420 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.280268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.9.231.126:33325)
0620 02:21:41.280560 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.280569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.280975 (+ 406us) server_negotiation.cc:407] Connection header received
0620 02:21:41.281215 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.281223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.281314 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.281421 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.282571 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.282576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.282579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.282596 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.282631 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.282644 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.282775 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.282940 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.283018 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 02:21:41.288151 14836 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:21:41.295496 14836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:41.318274 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.310679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:45065 (local address 127.0.0.1:45922)
0620 02:21:41.313989 (+ 3310us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.314031 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:41.314368 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:41.315097 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:41.315106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:41.315216 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:41.316276 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:41.316286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:41.316338 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:41.316477 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:41.317199 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:41.317217 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:41.317231 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 02:21:41.317479 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 02:21:41.317608 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3074,"thread_start_us":2907,"threads_started":1}
I20260620 02:21:41.319497 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.310848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.9.231.66:45065)
0620 02:21:41.311648 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.311659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.314305 (+ 2646us) server_negotiation.cc:407] Connection header received
0620 02:21:41.314533 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.314544 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.314839 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.315015 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.316756 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.316780 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.316804 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.316847 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.316932 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.316960 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.317126 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.318633 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.318821 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":406,"threads_started":1}
I20260620 02:21:41.320763 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.310904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38658)
0620 02:21:41.314937 (+ 4033us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.314957 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:41.315051 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:41.316183 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:41.316188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:41.316214 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:41.318622 (+ 2408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:41.318628 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:41.318656 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:41.318732 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:41.319550 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:41.319559 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:41.319566 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:21:41.320021 (+ 455us) client_negotiation.cc:240] Negotiation successful
0620 02:21:41.320118 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3886,"thread_start_us":3743,"threads_started":1}
I20260620 02:21:41.322288 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.311760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.9.231.65:40543)
0620 02:21:41.312382 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.312388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.315209 (+ 2821us) server_negotiation.cc:407] Connection header received
0620 02:21:41.315285 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.315292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.315498 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.315663 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.318995 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.319010 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.319035 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.319113 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.319208 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.319242 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.319472 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.321544 (+ 2072us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.321715 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":296,"threads_started":1}
I20260620 02:21:41.327097 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.315052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:38213 (local address 127.0.0.1:58070)
0620 02:21:41.320229 (+ 5177us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.320257 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:41.320402 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:41.324927 (+ 4525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:41.324933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:41.324985 (+ 52us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:41.325679 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:41.325683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:41.325706 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:41.325774 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:41.326580 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:41.326585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:41.326590 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:21:41.326702 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:21:41.326770 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5016,"threads_started":1}
I20260620 02:21:41.327258 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.316334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.9.231.67:38213)
0620 02:21:41.324376 (+ 8042us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.324382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.324413 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:21:41.324478 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.324484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.324678 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.324922 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.326375 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.326381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.326392 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.326404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.326441 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.326455 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.326657 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.326726 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.326821 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7870,"thread_start_us":7677,"threads_started":1}
I20260620 02:21:41.328456 14958 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f178ee115421484988074cdd247fc (DEFAULT_TABLE table=test-table [id=589510a35e07449689c8ec8af0774924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:41.330219 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f178ee115421484988074cdd247fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:41.332127 15089 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f178ee115421484988074cdd247fc (DEFAULT_TABLE table=test-table [id=589510a35e07449689c8ec8af0774924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:41.332058 15220 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f178ee115421484988074cdd247fc (DEFAULT_TABLE table=test-table [id=589510a35e07449689c8ec8af0774924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:41.333653 15220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f178ee115421484988074cdd247fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:41.333653 15089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f178ee115421484988074cdd247fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:41.349673 15305 tablet_bootstrap.cc:492] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Bootstrap starting.
I20260620 02:21:41.353309 15306 tablet_bootstrap.cc:492] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: Bootstrap starting.
I20260620 02:21:41.353377 15305 tablet_bootstrap.cc:654] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:41.355611 15305 log.cc:824] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:41.357393 15307 tablet_bootstrap.cc:492] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Bootstrap starting.
I20260620 02:21:41.357762 15306 tablet_bootstrap.cc:654] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:41.359757 15306 log.cc:824] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:41.360301 15305 tablet_bootstrap.cc:492] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: No bootstrap required, opened a new log
I20260620 02:21:41.360690 15305 ts_tablet_manager.cc:1397] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 02:21:41.361281 15307 tablet_bootstrap.cc:654] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:41.362684 15307 log.cc:824] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:41.363857 15306 tablet_bootstrap.cc:492] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: No bootstrap required, opened a new log
I20260620 02:21:41.364213 15306 ts_tablet_manager.cc:1397] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 02:21:41.366231 15307 tablet_bootstrap.cc:492] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: No bootstrap required, opened a new log
I20260620 02:21:41.366531 15307 ts_tablet_manager.cc:1397] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 02:21:41.370649 15305 raft_consensus.cc:348] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.371327 15305 raft_consensus.cc:374] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:41.371481 15305 raft_consensus.cc:729] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d665f99736c1400797f6cd5d0e1377a3, State: Initialized, Role: FOLLOWER
I20260620 02:21:41.372349 15305 consensus_queue.cc:260] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.373659 15287 heartbeater.cc:503] Master 127.9.231.126:33325 was elected leader, sending a full tablet report...
I20260620 02:21:41.374409 15305 ts_tablet_manager.cc:1428] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 02:21:41.376402 15306 raft_consensus.cc:348] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.376991 15306 raft_consensus.cc:374] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:41.377090 15306 raft_consensus.cc:729] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd99ed07865453bbc4a14606a4f95d0, State: Initialized, Role: FOLLOWER
I20260620 02:21:41.377811 15306 consensus_queue.cc:260] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.378715 15307 raft_consensus.cc:348] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.379244 15307 raft_consensus.cc:374] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:41.379364 15307 raft_consensus.cc:729] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2571c42a24ce4bdcbc66fca9749e1a7f, State: Initialized, Role: FOLLOWER
I20260620 02:21:41.380028 15306 ts_tablet_manager.cc:1428] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 02:21:41.380239 15307 consensus_queue.cc:260] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.381475 15155 heartbeater.cc:503] Master 127.9.231.126:33325 was elected leader, sending a full tablet report...
I20260620 02:21:41.382192 15307 ts_tablet_manager.cc:1428] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260620 02:21:41.499908 15156 tablet.cc:2369] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:21:41.504382 15288 tablet.cc:2369] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:21:41.512499 15024 tablet.cc:2369] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:41.543200 15313 raft_consensus.cc:482] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:41.543615 15313 raft_consensus.cc:504] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.545559 15313 leader_election.cc:288] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cd99ed07865453bbc4a14606a4f95d0 (127.9.231.65:40543), d665f99736c1400797f6cd5d0e1377a3 (127.9.231.67:38213)
I20260620 02:21:41.549566 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.545986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:41583 (local address 127.9.231.65:40543)
0620 02:21:41.546315 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.546325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.547368 (+ 1043us) server_negotiation.cc:407] Connection header received
0620 02:21:41.547578 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.547584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.547650 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.547767 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.548874 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.548880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.548890 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.548908 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.548940 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.548952 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.549181 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.549232 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.549296 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260620 02:21:41.549696 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.546251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:37857 (local address 127.9.231.67:38213)
0620 02:21:41.546507 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.546515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.547491 (+ 976us) server_negotiation.cc:407] Connection header received
0620 02:21:41.547708 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.547711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.547762 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.547906 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.549049 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.549056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.549065 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.549081 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.549112 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.549124 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.549266 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.549320 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.549387 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 02:21:41.549893 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.546038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:38213 (local address 127.9.231.66:37857)
0620 02:21:41.547362 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.547378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:41.547558 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:41.547925 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:41.547930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:41.547958 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:41.548839 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:41.548841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:41.548855 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:41.548891 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:41.549223 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:41.549228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:41.549233 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:21:41.549368 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 02:21:41.549418 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":982,"threads_started":1}
I20260620 02:21:41.550123 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.546005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.9.231.66:41583)
0620 02:21:41.547228 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.547247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:41.547432 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:41.547792 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:41.547816 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:41.547840 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:41.548647 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:21:41.548650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:41.548667 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:21:41.548709 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:41.549050 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:41.549055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:41.549060 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:21:41.549203 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:21:41.549266 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"thread_start_us":654,"threads_started":1}
I20260620 02:21:41.551280 15240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f178ee115421484988074cdd247fc" candidate_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d665f99736c1400797f6cd5d0e1377a3" is_pre_election: true
I20260620 02:21:41.551280 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f178ee115421484988074cdd247fc" candidate_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd99ed07865453bbc4a14606a4f95d0" is_pre_election: true
I20260620 02:21:41.551918 14978 raft_consensus.cc:2454] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2571c42a24ce4bdcbc66fca9749e1a7f in term 0.
I20260620 02:21:41.551918 15240 raft_consensus.cc:2454] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2571c42a24ce4bdcbc66fca9749e1a7f in term 0.
I20260620 02:21:41.552963 15042 leader_election.cc:302] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2571c42a24ce4bdcbc66fca9749e1a7f, d665f99736c1400797f6cd5d0e1377a3; no voters:
I20260620 02:21:41.553439 15313 raft_consensus.cc:2781] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:41.553634 15313 raft_consensus.cc:482] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:41.553758 15313 raft_consensus.cc:3037] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:41.556221 15313 raft_consensus.cc:504] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.556950 15313 leader_election.cc:288] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [CANDIDATE]: Term 1 election: Requested vote from peers 9cd99ed07865453bbc4a14606a4f95d0 (127.9.231.65:40543), d665f99736c1400797f6cd5d0e1377a3 (127.9.231.67:38213)
I20260620 02:21:41.557577 14978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f178ee115421484988074cdd247fc" candidate_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd99ed07865453bbc4a14606a4f95d0"
I20260620 02:21:41.557629 15240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f178ee115421484988074cdd247fc" candidate_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d665f99736c1400797f6cd5d0e1377a3"
I20260620 02:21:41.557808 14978 raft_consensus.cc:3037] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:41.557917 15240 raft_consensus.cc:3037] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:41.560417 14978 raft_consensus.cc:2454] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2571c42a24ce4bdcbc66fca9749e1a7f in term 1.
I20260620 02:21:41.560448 15240 raft_consensus.cc:2454] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2571c42a24ce4bdcbc66fca9749e1a7f in term 1.
I20260620 02:21:41.561043 15042 leader_election.cc:302] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2571c42a24ce4bdcbc66fca9749e1a7f, d665f99736c1400797f6cd5d0e1377a3; no voters:
I20260620 02:21:41.561467 15313 raft_consensus.cc:2781] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:41.562013 15313 raft_consensus.cc:686] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 1 LEADER]: Becoming Leader. State: Replica: 2571c42a24ce4bdcbc66fca9749e1a7f, State: Running, Role: LEADER
I20260620 02:21:41.562697 15313 consensus_queue.cc:237] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } }
I20260620 02:21:41.568189 14835 catalog_manager.cc:5495] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2571c42a24ce4bdcbc66fca9749e1a7f (127.9.231.66). New cstate: current_term: 1 leader_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571c42a24ce4bdcbc66fca9749e1a7f" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 } health_report { overall_health: UNKNOWN } } }
I20260620 02:21:41.584602 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.581845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.9.231.66:45065)
0620 02:21:41.582045 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.582049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.582108 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:21:41.582407 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.582412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.582502 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.582599 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.583910 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.583915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.583919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.583938 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.583975 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.583990 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.584104 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.584301 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.584355 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:21:41.594247 14978 raft_consensus.cc:1261] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0 [term 1 FOLLOWER]: Refusing update from remote peer 2571c42a24ce4bdcbc66fca9749e1a7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:41.594406 15240 raft_consensus.cc:1261] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 1 FOLLOWER]: Refusing update from remote peer 2571c42a24ce4bdcbc66fca9749e1a7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:41.595856 15316 consensus_queue.cc:1035] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d665f99736c1400797f6cd5d0e1377a3" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:41.596282 15313 consensus_queue.cc:1035] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd99ed07865453bbc4a14606a4f95d0" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:41.628880 14835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:41.629204 14835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:41.633144 14835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 953bd671bfbb4bc9a66fec7dd518f61e: Sending DeleteTablet for 3 replicas of tablet 2f6f178ee115421484988074cdd247fc
I20260620 02:21:41.634763 14958 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f178ee115421484988074cdd247fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:41 UTC) from {username='slave'} at 127.0.0.1:38658
I20260620 02:21:41.634763 15089 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f178ee115421484988074cdd247fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:41 UTC) from {username='slave'} at 127.0.0.1:45922
I20260620 02:21:41.635031 15220 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f178ee115421484988074cdd247fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:41 UTC) from {username='slave'} at 127.0.0.1:58070
I20260620 02:21:41.637315 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 14895
I20260620 02:21:41.664144 15331 tablet_replica.cc:330] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: stopping tablet replica
I20260620 02:21:41.665892 15331 raft_consensus.cc:2229] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:41.667490 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.638743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.9.231.65:40543)
0620 02:21:41.638937 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.638943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.662937 (+ 23994us) server_negotiation.cc:407] Connection header received
0620 02:21:41.664482 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.664490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.664633 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.664725 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.666580 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.666599 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.666604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.666629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.666673 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.666692 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.666806 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.667080 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.667158 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:21:41.669174 15018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:41.669612 15331 raft_consensus.cc:2258] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:41.672928 15331 ts_tablet_manager.cc:1905] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:41.681885 15331 ts_tablet_manager.cc:1918] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:41.682828 15329 tablet_replica.cc:330] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: stopping tablet replica
I20260620 02:21:41.683467 15329 raft_consensus.cc:2229] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:41.684197 15329 raft_consensus.cc:2258] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:41.684890 15331 log.cc:1196] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/2f6f178ee115421484988074cdd247fc
I20260620 02:21:41.685510 15331 ts_tablet_manager.cc:1939] T 2f6f178ee115421484988074cdd247fc P d665f99736c1400797f6cd5d0e1377a3: Deleting consensus metadata
I20260620 02:21:41.686968 15329 ts_tablet_manager.cc:1905] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:41.687659 14820 catalog_manager.cc:4841] TS d665f99736c1400797f6cd5d0e1377a3 (127.9.231.67:38213): tablet 2f6f178ee115421484988074cdd247fc (table test-table [id=589510a35e07449689c8ec8af0774924]) successfully deleted
I20260620 02:21:41.695475 15329 ts_tablet_manager.cc:1918] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:41.695809 15329 log.cc:1196] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/2f6f178ee115421484988074cdd247fc
I20260620 02:21:41.696470 15329 ts_tablet_manager.cc:1939] T 2f6f178ee115421484988074cdd247fc P 2571c42a24ce4bdcbc66fca9749e1a7f: Deleting consensus metadata
I20260620 02:21:41.698690 14820 catalog_manager.cc:4841] TS 2571c42a24ce4bdcbc66fca9749e1a7f (127.9.231.66:45065): tablet 2f6f178ee115421484988074cdd247fc (table test-table [id=589510a35e07449689c8ec8af0774924]) successfully deleted
I20260620 02:21:41.798648 15330 tablet_replica.cc:330] T 2f6f178ee115421484988074cdd247fc P 9cd99ed07865453bbc4a14606a4f95d0: stopping tablet replica
I20260620 02:21:41.798918 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 14895
W20260620 02:21:41.817729 14823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.231.65:40543 (error 108)
I20260620 02:21:41.818485 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15027
I20260620 02:21:41.819947 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.819246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38670)
0620 02:21:41.819539 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.819725 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260620 02:21:41.820204 14823 catalog_manager.cc:4584] TS 9cd99ed07865453bbc4a14606a4f95d0 (127.9.231.65:40543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f6f178ee115421484988074cdd247fc: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
I20260620 02:21:41.821934 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.819321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.9.231.66:45065)
0620 02:21:41.819504 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.819511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.819652 (+ 141us) server_negotiation.cc:407] Connection header received
0620 02:21:41.819887 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.819893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.819959 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.820080 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.821212 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.821219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.821223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.821245 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.821287 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.821299 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.821438 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.821544 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.821617 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:21:41.822681 15150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:41.874246 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.873670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38678)
0620 02:21:41.873916 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.874014 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 02:21:41.875574 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.875210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38690)
0620 02:21:41.875355 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.875431 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 02:21:41.935624 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15027
I20260620 02:21:41.937139 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.935927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38704)
0620 02:21:41.936817 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.936934 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":717}
I20260620 02:21:41.938558 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.938066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38716)
0620 02:21:41.938258 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:41.938346 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:21:41.956491 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15158
I20260620 02:21:41.959517 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:41.957339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.9.231.67:38213)
0620 02:21:41.957531 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:41.957535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:41.957556 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:21:41.957727 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:41.957731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:41.957827 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:41.957918 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:41.958860 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:41.958864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:41.958867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:41.958881 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:41.958912 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:41.958923 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:41.959029 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:41.959222 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 02:21:41.959273 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:21:41.960206 15282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:42.019498 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:42.018825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38730)
0620 02:21:42.019224 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:42.019332 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 02:21:42.020795 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:42.020345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40543 (local address 127.0.0.1:38744)
0620 02:21:42.020548 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:42.020636 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 02:21:42.055240 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15158
I20260620 02:21:42.072466 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 14805
I20260620 02:21:42.073761 14866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:42.183888 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 14805
2026-06-20T02:21:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3067 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 02:21:42.229894 10141 mini_oidc.cc:152] Starting JWKS server
I20260620 02:21:42.233062 10141 webserver.cc:460] Webserver started at https://127.0.0.1:39125/ using document root <none> and password file <none>
I20260620 02:21:42.233538 10141 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:21:42.234342 10141 webserver.cc:460] Webserver started at http://127.0.0.1:42861/ using document root <none> and password file <none>
2026-06-20T02:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:42Z Disabled control of system clock
I20260620 02:21:42.265043 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34565
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34565
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/testchainca.cert with env {}
W20260620 02:21:42.613652 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:42.614061 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:42.614240 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:42.622069 15343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:42.622272 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:42.622371 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:42.622478 15343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:42.622565 15343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:21:42.622658 15343 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:21:42.622736 15343 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39125/jwks.json
I20260620 02:21:42.633845 15343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:42.635504 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:42.636925 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:42.650496 15348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:42.650425 15349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:42.650398 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:42.651578 15343 server_base.cc:1034] running on GCE node
I20260620 02:21:42.652611 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:42.654335 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:42.655705 15343 hybrid_clock.cc:648] HybridClock initialized: now 1781922102655654 us; error 60 us; skew 500 ppm
I20260620 02:21:42.656158 15343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:42.658952 15343 webserver.cc:460] Webserver started at http://127.9.231.126:32935/ using document root <none> and password file <none>
I20260620 02:21:42.659816 15343 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:42.659976 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:42.660384 15343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:42.663980 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "5d777ce581ce4795b548cd1a654f6e86"
format_stamp: "Formatted at 2026-06-20 02:21:42 on dist-test-slave-9vg1"
server_key: "fd5180bf3e4eb198d451df2ca812e20b"
server_key_iv: "c24daa9e56bd7bab27ee515d9d2c1b03"
server_key_version: "encryptionkey@0"
I20260620 02:21:42.664956 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "5d777ce581ce4795b548cd1a654f6e86"
format_stamp: "Formatted at 2026-06-20 02:21:42 on dist-test-slave-9vg1"
server_key: "fd5180bf3e4eb198d451df2ca812e20b"
server_key_iv: "c24daa9e56bd7bab27ee515d9d2c1b03"
server_key_version: "encryptionkey@0"
I20260620 02:21:42.671239 15343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:21:42.675391 15357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:42.677065 15343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:42.677312 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "5d777ce581ce4795b548cd1a654f6e86"
format_stamp: "Formatted at 2026-06-20 02:21:42 on dist-test-slave-9vg1"
server_key: "fd5180bf3e4eb198d451df2ca812e20b"
server_key_iv: "c24daa9e56bd7bab27ee515d9d2c1b03"
server_key_version: "encryptionkey@0"
I20260620 02:21:42.677590 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:42.700871 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:42.701941 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:42.702314 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:42.724877 15343 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34565
I20260620 02:21:42.724902 15408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34565 every 8 connection(s)
I20260620 02:21:42.726487 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:42.727810 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15343
I20260620 02:21:42.728019 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:42.728847 10141 external_mini_cluster.cc:1428] Setting key d77baa9514649bb2fe7bf5068238c821
I20260620 02:21:42.732028 15409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:42.744125 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:42.731042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.9.231.126:34565)
0620 02:21:42.732410 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:42.732423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:42.732471 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:21:42.734193 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:42.734219 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:42.734817 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:42.735369 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:42.736026 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:42.737525 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:42.738928 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:42.739739 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:42.739964 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:42.741321 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:42.741330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:42.741346 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:42.741984 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:42.742068 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:42.742100 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:42.742289 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:42.742566 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 02:21:42.742986 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":381,"threads_started":1}
I20260620 02:21:42.744860 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86: Bootstrap starting.
I20260620 02:21:42.748558 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:21:42.748906 15409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:42.750063 15409 log.cc:824] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:42.753163 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86: No bootstrap required, opened a new log
I20260620 02:21:42.761018 15409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d777ce581ce4795b548cd1a654f6e86" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34565 } }
I20260620 02:21:42.761484 15409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:42.761605 15409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d777ce581ce4795b548cd1a654f6e86, State: Initialized, Role: FOLLOWER
I20260620 02:21:42.762306 15409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d777ce581ce4795b548cd1a654f6e86" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34565 } }
I20260620 02:21:42.762583 15409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:42.762734 15409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:42.762924 15409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:42.765429 15409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d777ce581ce4795b548cd1a654f6e86" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34565 } }
I20260620 02:21:42.766060 15409 leader_election.cc:302] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d777ce581ce4795b548cd1a654f6e86; no voters:
I20260620 02:21:42.766711 15409 leader_election.cc:288] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:42.766923 15414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:42.767859 15414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [term 1 LEADER]: Becoming Leader. State: Replica: 5d777ce581ce4795b548cd1a654f6e86, State: Running, Role: LEADER
I20260620 02:21:42.768462 15414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d777ce581ce4795b548cd1a654f6e86" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34565 } }
I20260620 02:21:42.769361 15409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:42.771839 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d777ce581ce4795b548cd1a654f6e86. Latest consensus state: current_term: 1 leader_uuid: "5d777ce581ce4795b548cd1a654f6e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d777ce581ce4795b548cd1a654f6e86" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34565 } } }
I20260620 02:21:42.772161 15415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d777ce581ce4795b548cd1a654f6e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d777ce581ce4795b548cd1a654f6e86" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34565 } } }
I20260620 02:21:42.772450 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:42.772468 15415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:42.773924 15422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:42.780958 15422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:42.791514 15422 catalog_manager.cc:1347] Generated new cluster ID: e726b345af2040afb2a412bb5152382b
I20260620 02:21:42.791666 15422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:42.810967 15422 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:42.812669 15422 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:42.826596 15422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d777ce581ce4795b548cd1a654f6e86: Generated new TSK 0
I20260620 02:21:42.827668 15422 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:21:42.878598 15440 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 02:21:42.878782 15440 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:21:42.880079 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:42.850703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.9.231.126:34565)
0620 02:21:42.850959 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:42.850970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:42.851713 (+ 743us) server_negotiation.cc:407] Connection header received
0620 02:21:42.852160 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:42.852165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:42.852258 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:42.852438 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:21:42.853116 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:42.853948 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:42.855650 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:42.856183 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:42.856362 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:42.857015 (+ 653us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:21:42.879308 (+ 22293us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:21:42.879742 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 02:21:42.880220 15412 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 02:21:42.881726 15439 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:42.850780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34565 (local address 127.0.0.1:39996)
0620 02:21:42.851567 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:42.851599 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:42.851852 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:42.852466 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:42.852471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:42.852485 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:21:42.852913 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:42.852923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:42.854120 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:42.854129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:42.855456 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:42.855498 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:42.856420 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:42.856426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:42.856612 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:42.856684 (+ 72us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:21:42.879667 (+ 22983us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:21:42.879980 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.231.126:34565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":424,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.231.126:34565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 02:21:42.883975 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15343
I20260620 02:21:42.885363 15404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:42.983520 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15343
2026-06-20T02:21:43Z chronyd exiting
I20260620 02:21:43.036307 10141 test_util.cc:175] -----------------------------------------------
I20260620 02:21:43.036442 10141 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922055280103-10141-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (811 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 02:21:43.040514 10141 mini_oidc.cc:152] Starting JWKS server
I20260620 02:21:43.042409 10141 webserver.cc:460] Webserver started at https://127.0.0.1:44741/ using document root <none> and password file <none>
I20260620 02:21:43.042678 10141 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:21:43.043442 10141 webserver.cc:460] Webserver started at http://127.0.0.1:37935/ using document root <none> and password file <none>
2026-06-20T02:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:43Z Disabled control of system clock
I20260620 02:21:43.073648 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34063
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44741/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/testchainca.cert with env {}
W20260620 02:21:43.440160 15447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:43.440554 15447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:43.440714 15447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:43.449328 15447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:43.449541 15447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:43.449644 15447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:43.449764 15447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:43.449860 15447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:21:43.449963 15447 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:21:43.450054 15447 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44741/jwks.json
I20260620 02:21:43.461187 15447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44741/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:43.462687 15447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:43.464175 15447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:43.477588 15452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:43.477624 15455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:43.477548 15453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:43.478506 15447 server_base.cc:1034] running on GCE node
I20260620 02:21:43.479425 15447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:43.480890 15447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:43.482277 15447 hybrid_clock.cc:648] HybridClock initialized: now 1781922103482240 us; error 44 us; skew 500 ppm
I20260620 02:21:43.482692 15447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:43.485455 15447 webserver.cc:460] Webserver started at http://127.9.231.126:39371/ using document root <none> and password file <none>
I20260620 02:21:43.486256 15447 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:43.486403 15447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:43.486752 15447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:43.489885 15447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "f8ce9ecbe5304924a3c84b4db8f2180a"
format_stamp: "Formatted at 2026-06-20 02:21:43 on dist-test-slave-9vg1"
server_key: "69e74e5c413162a379aec5db8a84d6d6"
server_key_iv: "4e459eaaa9ad1c92d174744b1ac771e9"
server_key_version: "encryptionkey@0"
I20260620 02:21:43.490780 15447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "f8ce9ecbe5304924a3c84b4db8f2180a"
format_stamp: "Formatted at 2026-06-20 02:21:43 on dist-test-slave-9vg1"
server_key: "69e74e5c413162a379aec5db8a84d6d6"
server_key_iv: "4e459eaaa9ad1c92d174744b1ac771e9"
server_key_version: "encryptionkey@0"
I20260620 02:21:43.496599 15447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:21:43.500499 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:43.502216 15447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:43.502460 15447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "f8ce9ecbe5304924a3c84b4db8f2180a"
format_stamp: "Formatted at 2026-06-20 02:21:43 on dist-test-slave-9vg1"
server_key: "69e74e5c413162a379aec5db8a84d6d6"
server_key_iv: "4e459eaaa9ad1c92d174744b1ac771e9"
server_key_version: "encryptionkey@0"
I20260620 02:21:43.502748 15447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:43.519836 15447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:43.520808 15447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:43.521165 15447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:43.541332 15447 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34063
I20260620 02:21:43.541357 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34063 every 8 connection(s)
I20260620 02:21:43.542908 15447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:43.547623 15513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:43.548759 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15447
I20260620 02:21:43.549011 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:43.549701 10141 external_mini_cluster.cc:1428] Setting key 43cd64766b1b48895384eff1a0aefcfc
I20260620 02:21:43.561698 15513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a: Bootstrap starting.
I20260620 02:21:43.564363 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:43.551689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.9.231.126:34063)
0620 02:21:43.552806 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:43.552821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:43.552867 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:21:43.554618 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:43.554645 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:43.555312 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:43.555756 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:43.556360 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:43.557697 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:43.559012 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:43.559823 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:43.560021 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:43.561474 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:43.561488 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:43.561510 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:43.562427 (+ 917us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:43.562541 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:43.562573 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:43.562723 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:43.562981 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 02:21:43.563365 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":376,"threads_started":1}
I20260620 02:21:43.565783 15513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:43.566962 15513 log.cc:824] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:43.568754 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:21:43.570083 15513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a: No bootstrap required, opened a new log
I20260620 02:21:43.577988 15513 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34063 } }
I20260620 02:21:43.578449 15513 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:43.578558 15513 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ce9ecbe5304924a3c84b4db8f2180a, State: Initialized, Role: FOLLOWER
I20260620 02:21:43.579234 15513 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34063 } }
I20260620 02:21:43.579488 15513 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:43.579634 15513 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:43.579810 15513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:43.582358 15513 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34063 } }
I20260620 02:21:43.582924 15513 leader_election.cc:302] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8ce9ecbe5304924a3c84b4db8f2180a; no voters:
I20260620 02:21:43.583664 15513 leader_election.cc:288] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:43.583884 15518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:43.584702 15518 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [term 1 LEADER]: Becoming Leader. State: Replica: f8ce9ecbe5304924a3c84b4db8f2180a, State: Running, Role: LEADER
I20260620 02:21:43.585282 15518 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34063 } }
I20260620 02:21:43.586054 15513 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:43.588491 15519 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34063 } } }
I20260620 02:21:43.589004 15519 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:43.589327 15520 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ce9ecbe5304924a3c84b4db8f2180a. Latest consensus state: current_term: 1 leader_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ce9ecbe5304924a3c84b4db8f2180a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34063 } } }
I20260620 02:21:43.589668 15520 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:43.592204 15526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:43.598387 15526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:43.608685 15526 catalog_manager.cc:1347] Generated new cluster ID: 1d6641dd304e4036a2a9f2645ac5dfef
I20260620 02:21:43.608855 15526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:43.624773 15526 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:43.626634 15526 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:43.634756 15526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8ce9ecbe5304924a3c84b4db8f2180a: Generated new TSK 0
I20260620 02:21:43.635798 15526 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:21:43.672238 15543 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 02:21:43.672401 15543 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:21:43.673666 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:43.651538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.9.231.126:34063)
0620 02:21:43.651759 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:43.651767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:43.652234 (+ 467us) server_negotiation.cc:407] Connection header received
0620 02:21:43.652492 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:43.652498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:43.652625 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:43.652814 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:21:43.653386 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:43.654538 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:43.656128 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:43.656874 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:43.657026 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:43.657646 (+ 620us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:21:43.672883 (+ 15237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:21:43.673333 (+ 450us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49694: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 02:21:43.673780 15542 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:43.651393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34063 (local address 127.0.0.1:49694)
0620 02:21:43.652108 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:43.652125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:43.652276 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:43.652858 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:43.652862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:43.652874 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:21:43.653213 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:43.653223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:43.654735 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:43.654739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:43.655952 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:43.655968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:43.657080 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:43.657086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:43.657234 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:43.657281 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:21:43.673187 (+ 15906us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:21:43.673422 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.231.126:34063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":333,"threads_started":1}
W20260620 02:21:43.673799 15516 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49694: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 02:21:43.675877 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15447
I20260620 02:21:43.677204 15508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:43.773525 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15447
2026-06-20T02:21:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (790 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 02:21:43.831334 10141 mini_oidc.cc:152] Starting JWKS server
I20260620 02:21:43.833467 10141 webserver.cc:460] Webserver started at https://127.0.0.1:39481/ using document root <none> and password file <none>
I20260620 02:21:43.833766 10141 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:21:43.834528 10141 webserver.cc:460] Webserver started at http://127.0.0.1:42561/ using document root <none> and password file <none>
2026-06-20T02:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:43Z Disabled control of system clock
I20260620 02:21:43.865804 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:39297
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:39297
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39481/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:21:44.224700 15550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:44.225047 15550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:44.225147 15550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:44.232908 15550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:44.233042 15550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:44.233080 15550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:44.233211 15550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:44.233320 15550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:21:44.233438 15550 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:21:44.233498 15550 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39481/jwks.json
I20260620 02:21:44.243363 15550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:39297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39481/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:44.244908 15550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:44.246912 15550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:44.260599 15556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:44.260612 15558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:44.260679 15555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:44.261507 15550 server_base.cc:1034] running on GCE node
I20260620 02:21:44.262478 15550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:44.264355 15550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:44.265641 15550 hybrid_clock.cc:648] HybridClock initialized: now 1781922104265578 us; error 45 us; skew 500 ppm
I20260620 02:21:44.266045 15550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:44.268635 15550 webserver.cc:460] Webserver started at http://127.9.231.126:37605/ using document root <none> and password file <none>
I20260620 02:21:44.269455 15550 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:44.269608 15550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:44.269968 15550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:44.273351 15550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "fc0900caaac046068e3080e33316150a"
format_stamp: "Formatted at 2026-06-20 02:21:44 on dist-test-slave-9vg1"
server_key: "5258ac0c45ce7cfd32cbd1d1bbc20a20"
server_key_iv: "39ee80094f4a0b60f05e65b7b6ebf527"
server_key_version: "encryptionkey@0"
I20260620 02:21:44.274299 15550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "fc0900caaac046068e3080e33316150a"
format_stamp: "Formatted at 2026-06-20 02:21:44 on dist-test-slave-9vg1"
server_key: "5258ac0c45ce7cfd32cbd1d1bbc20a20"
server_key_iv: "39ee80094f4a0b60f05e65b7b6ebf527"
server_key_version: "encryptionkey@0"
I20260620 02:21:44.280252 15550 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:21:44.284631 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:44.286459 15550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260620 02:21:44.286720 15550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "fc0900caaac046068e3080e33316150a"
format_stamp: "Formatted at 2026-06-20 02:21:44 on dist-test-slave-9vg1"
server_key: "5258ac0c45ce7cfd32cbd1d1bbc20a20"
server_key_iv: "39ee80094f4a0b60f05e65b7b6ebf527"
server_key_version: "encryptionkey@0"
I20260620 02:21:44.287010 15550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:44.302788 15550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:44.303800 15550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:44.304164 15550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:44.324894 15550 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:39297
I20260620 02:21:44.324916 15615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:39297 every 8 connection(s)
I20260620 02:21:44.326396 15550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:44.330090 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15550
I20260620 02:21:44.330305 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:44.330770 15616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:44.331034 10141 external_mini_cluster.cc:1428] Setting key 787286266fe456d718e1fbfb91e8200a
I20260620 02:21:44.346150 15616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a: Bootstrap starting.
I20260620 02:21:44.346704 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:44.333172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.9.231.126:39297)
0620 02:21:44.334306 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:44.334319 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:44.334362 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:21:44.336289 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:44.336316 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:44.336995 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:44.337410 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:44.338073 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:44.339582 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:44.341437 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:44.342166 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:44.342322 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:44.343821 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:44.343830 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:44.343847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:44.344494 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:44.344573 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:44.344597 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:44.344691 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:44.345107 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 02:21:44.345466 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":407,"threads_started":1}
I20260620 02:21:44.350111 15616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:44.351480 15616 log.cc:824] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:44.351601 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:21:44.355052 15616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a: No bootstrap required, opened a new log
I20260620 02:21:44.363162 15616 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0900caaac046068e3080e33316150a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39297 } }
I20260620 02:21:44.363654 15616 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:44.363782 15616 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0900caaac046068e3080e33316150a, State: Initialized, Role: FOLLOWER
I20260620 02:21:44.364475 15616 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0900caaac046068e3080e33316150a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39297 } }
I20260620 02:21:44.364748 15616 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:44.364938 15616 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:44.365120 15616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:44.368106 15616 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0900caaac046068e3080e33316150a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39297 } }
I20260620 02:21:44.368745 15616 leader_election.cc:302] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0900caaac046068e3080e33316150a; no voters:
I20260620 02:21:44.369484 15616 leader_election.cc:288] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:44.369659 15621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:44.371181 15621 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [term 1 LEADER]: Becoming Leader. State: Replica: fc0900caaac046068e3080e33316150a, State: Running, Role: LEADER
I20260620 02:21:44.371829 15621 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0900caaac046068e3080e33316150a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39297 } }
I20260620 02:21:44.372061 15616 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:44.375207 15622 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0900caaac046068e3080e33316150a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0900caaac046068e3080e33316150a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39297 } } }
I20260620 02:21:44.375095 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0900caaac046068e3080e33316150a. Latest consensus state: current_term: 1 leader_uuid: "fc0900caaac046068e3080e33316150a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0900caaac046068e3080e33316150a" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39297 } } }
I20260620 02:21:44.375795 15622 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:44.375886 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:44.388965 15630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:44.395018 15630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:44.405699 15630 catalog_manager.cc:1347] Generated new cluster ID: 2b94015f0b5743289be8ceef62886dc2
I20260620 02:21:44.405844 15630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:44.446923 15630 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:44.449040 15630 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:44.461251 15630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc0900caaac046068e3080e33316150a: Generated new TSK 0
I20260620 02:21:44.462284 15630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:21:44.497663 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:44.487489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.9.231.126:39297)
0620 02:21:44.487742 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:44.487750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:44.488348 (+ 598us) server_negotiation.cc:407] Connection header received
0620 02:21:44.488532 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:44.488538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:44.488640 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:44.488840 (+ 200us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:21:44.489604 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:44.490813 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:44.492571 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:44.493458 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:44.493631 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:44.494326 (+ 695us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:21:44.496926 (+ 2600us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:21:44.497306 (+ 380us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 02:21:44.497800 15619 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 02:21:44.499783 15646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:44.487355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39297 (local address 127.0.0.1:53250)
0620 02:21:44.488098 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:44.488117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:44.488274 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:44.488918 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:44.488928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:44.488994 (+ 66us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:21:44.489374 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:44.489385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:44.490999 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:44.491003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:44.492384 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:44.492399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:44.493735 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:44.493741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:44.493920 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:44.493966 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:21:44.497209 (+ 3243us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:21:44.499333 (+ 2124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.231.126:39297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1728,"thread_start_us":414,"threads_started":1}
I20260620 02:21:44.502076 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15550
I20260620 02:21:44.503557 15611 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 02:21:44.510232 15647 webserver.cc:533] Webserver: sslize failed:
E20260620 02:21:44.510383 15647 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:21:44.601135 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15550
2026-06-20T02:21:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (827 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T02:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:44Z Disabled control of system clock
I20260620 02:21:44.689954 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34881
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:46263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34881
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/testchainca.cert with env {}
W20260620 02:21:45.051823 15652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:45.052152 15652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:45.052296 15652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:45.060194 15652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:45.060384 15652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:45.060427 15652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:45.060505 15652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:45.060587 15652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:21:45.060691 15652 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:21:45.060760 15652 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 02:21:45.071445 15652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:46263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:45.072960 15652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:45.074533 15652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:45.087683 15657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:45.087664 15660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:45.087713 15658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:45.089306 15652 server_base.cc:1034] running on GCE node
I20260620 02:21:45.090349 15652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:45.092308 15652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:45.093636 15652 hybrid_clock.cc:648] HybridClock initialized: now 1781922105093559 us; error 77 us; skew 500 ppm
I20260620 02:21:45.094065 15652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:45.104441 15652 webserver.cc:460] Webserver started at http://127.9.231.126:37693/ using document root <none> and password file <none>
I20260620 02:21:45.105270 15652 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:45.105381 15652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:45.105654 15652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:45.108901 15652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "508c62a1a7d74b9e9ebcf5436396a6d9"
format_stamp: "Formatted at 2026-06-20 02:21:45 on dist-test-slave-9vg1"
server_key: "65a5763808fe68a21b768ec5245a2d34"
server_key_iv: "2486ae8f23da3c646947d34fb82e207f"
server_key_version: "encryptionkey@0"
I20260620 02:21:45.109745 15652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "508c62a1a7d74b9e9ebcf5436396a6d9"
format_stamp: "Formatted at 2026-06-20 02:21:45 on dist-test-slave-9vg1"
server_key: "65a5763808fe68a21b768ec5245a2d34"
server_key_iv: "2486ae8f23da3c646947d34fb82e207f"
server_key_version: "encryptionkey@0"
I20260620 02:21:45.115394 15652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:21:45.119298 15666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:45.120884 15652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:45.121112 15652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "508c62a1a7d74b9e9ebcf5436396a6d9"
format_stamp: "Formatted at 2026-06-20 02:21:45 on dist-test-slave-9vg1"
server_key: "65a5763808fe68a21b768ec5245a2d34"
server_key_iv: "2486ae8f23da3c646947d34fb82e207f"
server_key_version: "encryptionkey@0"
I20260620 02:21:45.121397 15652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:45.145579 15652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:45.146549 15652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:45.146878 15652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:45.168623 15652 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34881
I20260620 02:21:45.168641 15717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34881 every 8 connection(s)
I20260620 02:21:45.170140 15652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:45.174263 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15652
I20260620 02:21:45.174486 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:45.175201 10141 external_mini_cluster.cc:1428] Setting key 4f8f5c1222d44288315ca4ef0e70071e
I20260620 02:21:45.175308 15718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:45.191732 15718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9: Bootstrap starting.
I20260620 02:21:45.191995 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:45.177311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.9.231.126:34881)
0620 02:21:45.178453 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:45.178469 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:45.178525 (+ 56us) server_negotiation.cc:407] Connection header received
0620 02:21:45.180585 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:45.180617 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:45.181326 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:45.181856 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:45.182460 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:45.184525 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:45.185993 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:45.187104 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:45.187296 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:45.188807 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:45.188819 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:45.188848 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:21:45.189621 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:45.189732 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:21:45.189760 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:45.189881 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:45.190211 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:21:45.190589 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":351,"threads_started":1}
I20260620 02:21:45.195396 15718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:45.196138 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:21:45.196593 15718 log.cc:824] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:45.200670 15718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9: No bootstrap required, opened a new log
I20260620 02:21:45.208910 15718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34881 } }
I20260620 02:21:45.209316 15718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:45.209460 15718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508c62a1a7d74b9e9ebcf5436396a6d9, State: Initialized, Role: FOLLOWER
I20260620 02:21:45.210142 15718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34881 } }
I20260620 02:21:45.210364 15718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:45.210527 15718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:45.210717 15718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:45.213475 15718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34881 } }
I20260620 02:21:45.214061 15718 leader_election.cc:302] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 508c62a1a7d74b9e9ebcf5436396a6d9; no voters:
I20260620 02:21:45.214700 15718 leader_election.cc:288] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:45.214897 15724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:45.215698 15724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [term 1 LEADER]: Becoming Leader. State: Replica: 508c62a1a7d74b9e9ebcf5436396a6d9, State: Running, Role: LEADER
I20260620 02:21:45.216295 15724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34881 } }
I20260620 02:21:45.217052 15718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:45.219173 15725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34881 } } }
I20260620 02:21:45.219617 15725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:45.221673 15726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 508c62a1a7d74b9e9ebcf5436396a6d9. Latest consensus state: current_term: 1 leader_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508c62a1a7d74b9e9ebcf5436396a6d9" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34881 } } }
I20260620 02:21:45.222112 15726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:45.223526 15731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:45.230515 15731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:45.241598 15731 catalog_manager.cc:1347] Generated new cluster ID: cfe9db96ecab48f9b8391121203be82d
I20260620 02:21:45.241762 15731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:45.280109 15731 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:45.282001 15731 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:45.291379 15731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 508c62a1a7d74b9e9ebcf5436396a6d9: Generated new TSK 0
I20260620 02:21:45.292558 15731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:21:45.758375 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:45.319753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.9.231.126:34881)
0620 02:21:45.319993 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:45.320002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:45.320477 (+ 475us) server_negotiation.cc:407] Connection header received
0620 02:21:45.320839 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:45.320846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:45.320954 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:45.321159 (+ 205us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:21:45.321713 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:45.322879 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:45.324667 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:45.325446 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:45.325608 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:45.326230 (+ 622us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:21:45.757788 (+431558us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:21:45.758153 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47150: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:21:45.758505 15721 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47150: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 02:21:45.758844 15748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:45.319637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34881 (local address 127.0.0.1:47150)
0620 02:21:45.320353 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:45.320372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:45.320588 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:45.321243 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:45.321249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:45.321263 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:21:45.321544 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:45.321552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:45.323128 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:45.323139 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:45.324487 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:45.324501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:45.325704 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:45.325713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:45.325860 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:45.325899 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:21:45.758109 (+432210us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:21:45.758381 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.231.126:34881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":365,"threads_started":1}
I20260620 02:21:45.760859 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15652
I20260620 02:21:45.762146 15713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:45.868065 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15652
2026-06-20T02:21:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1247 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (442 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (450 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:46 dist-test-slave-9vg1 krb5kdc[15763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:46 dist-test-slave-9vg1 krb5kdc[15763](info): set up 2 sockets
Jun 20 02:21:46 dist-test-slave-9vg1 krb5kdc[15763](info): commencing operation
krb5kdc: starting...
W20260620 02:21:47.879218 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:48 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:21:48.115500 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40923
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:40923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:48.463366 15776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:48.463691 15776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:48.463783 15776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:48.471561 15776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:48.471704 15776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:48.471752 15776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:48.471803 15776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:48.471844 15776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:48.482702 15776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:40923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:48.484228 15776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:48.485630 15776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:48.498798 15781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:48.498927 15782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:48.498824 15784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:48.499738 15776 server_base.cc:1034] running on GCE node
I20260620 02:21:48.500716 15776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:48.502355 15776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:48.503582 15776 hybrid_clock.cc:648] HybridClock initialized: now 1781922108503530 us; error 39 us; skew 500 ppm
I20260620 02:21:48.503998 15776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:48 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922108, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:48.507440 15776 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:21:48.509244 15776 webserver.cc:460] Webserver started at http://127.9.231.126:35407/ using document root <none> and password file <none>
I20260620 02:21:48.509948 15776 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:48.510058 15776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:48.510337 15776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:48.513574 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "f615be5c9b5c43339d97e73cad0404ee"
format_stamp: "Formatted at 2026-06-20 02:21:48 on dist-test-slave-9vg1"
server_key: "356a8e08d42b7933cd84a06adf1ae0a1"
server_key_iv: "72768dbe6973604f2d3b243f879ba1bf"
server_key_version: "encryptionkey@0"
I20260620 02:21:48.514384 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "f615be5c9b5c43339d97e73cad0404ee"
format_stamp: "Formatted at 2026-06-20 02:21:48 on dist-test-slave-9vg1"
server_key: "356a8e08d42b7933cd84a06adf1ae0a1"
server_key_iv: "72768dbe6973604f2d3b243f879ba1bf"
server_key_version: "encryptionkey@0"
I20260620 02:21:48.520256 15776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:21:48.524483 15791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:48.526294 15776 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:21:48.526487 15776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "f615be5c9b5c43339d97e73cad0404ee"
format_stamp: "Formatted at 2026-06-20 02:21:48 on dist-test-slave-9vg1"
server_key: "356a8e08d42b7933cd84a06adf1ae0a1"
server_key_iv: "72768dbe6973604f2d3b243f879ba1bf"
server_key_version: "encryptionkey@0"
I20260620 02:21:48.526690 15776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:48.545449 15776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:48.550838 15776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:48.551373 15776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:48.572141 15776 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:40923
I20260620 02:21:48.572194 15842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:40923 every 8 connection(s)
I20260620 02:21:48.573668 15776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:48.577762 15843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:48.578379 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15776
I20260620 02:21:48.578641 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:48.579504 10141 external_mini_cluster.cc:1428] Setting key 1f40a422fe015319e7ae8a40f530ca8b
I20260620 02:21:48.593474 15843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee: Bootstrap starting.
Jun 20 02:21:48 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:48.597677 15843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:48.599287 15843 log.cc:824] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:48.603099 15843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee: No bootstrap required, opened a new log
I20260620 02:21:48.605921 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:48.581740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.9.231.126:40923)
0620 02:21:48.582558 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:48.582570 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:48.582611 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:21:48.584928 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:48.584959 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:48.585439 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:48.585810 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:48.587731 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:48.589081 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:48.590946 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:48.591807 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:48.591957 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:48.596789 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:48.596802 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:48.596822 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:48.596886 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:48.600778 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:48.601731 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:48.601743 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:48.601753 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:48.601977 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:48.602450 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:48.602455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:48.602458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:48.603224 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:48.603536 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:48.603990 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 02:21:48.604452 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":383,"threads_started":1}
I20260620 02:21:48.612342 15843 raft_consensus.cc:348] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f615be5c9b5c43339d97e73cad0404ee" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40923 } }
I20260620 02:21:48.612732 15843 raft_consensus.cc:374] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:48.612896 15843 raft_consensus.cc:729] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f615be5c9b5c43339d97e73cad0404ee, State: Initialized, Role: FOLLOWER
I20260620 02:21:48.613618 15843 consensus_queue.cc:260] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f615be5c9b5c43339d97e73cad0404ee" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40923 } }
I20260620 02:21:48.613880 15843 raft_consensus.cc:388] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:48.614063 15843 raft_consensus.cc:482] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:48.614359 15843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:48.616761 15843 raft_consensus.cc:504] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f615be5c9b5c43339d97e73cad0404ee" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40923 } }
I20260620 02:21:48.617326 15843 leader_election.cc:302] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f615be5c9b5c43339d97e73cad0404ee; no voters:
I20260620 02:21:48.617985 15843 leader_election.cc:288] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:48.618219 15848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:48.618947 15848 raft_consensus.cc:686] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [term 1 LEADER]: Becoming Leader. State: Replica: f615be5c9b5c43339d97e73cad0404ee, State: Running, Role: LEADER
I20260620 02:21:48.619764 15848 consensus_queue.cc:237] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f615be5c9b5c43339d97e73cad0404ee" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40923 } }
I20260620 02:21:48.620421 15843 sys_catalog.cc:564] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:48.622726 15850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f615be5c9b5c43339d97e73cad0404ee. Latest consensus state: current_term: 1 leader_uuid: "f615be5c9b5c43339d97e73cad0404ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f615be5c9b5c43339d97e73cad0404ee" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40923 } } }
I20260620 02:21:48.623165 15850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:48.623806 15849 sys_catalog.cc:455] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f615be5c9b5c43339d97e73cad0404ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f615be5c9b5c43339d97e73cad0404ee" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40923 } } }
I20260620 02:21:48.624125 15849 sys_catalog.cc:458] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:48.624286 15855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:48.632645 15855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:48.643436 15855 catalog_manager.cc:1347] Generated new cluster ID: 73375425797a4427a726a1d620762782
I20260620 02:21:48.643647 15855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:48.665120 15855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:48.666608 15855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:48.674826 15855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f615be5c9b5c43339d97e73cad0404ee: Generated new TSK 0
I20260620 02:21:48.675908 15855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:21:48.751853 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40923
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:49.112671 15871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:49.112982 15871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:49.113080 15871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:49.120895 15871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:49.121083 15871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:49.121217 15871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:49.132500 15871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:49.133908 15871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:49.135347 15871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:49.150035 15877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:49.150249 15879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:49.150519 15876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:49.151536 15871 server_base.cc:1034] running on GCE node
I20260620 02:21:49.152249 15871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:49.153362 15871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:49.154625 15871 hybrid_clock.cc:648] HybridClock initialized: now 1781922109154584 us; error 39 us; skew 500 ppm
I20260620 02:21:49.155020 15871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:49 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922109, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:49.158645 15871 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:21:49.160363 15871 webserver.cc:460] Webserver started at http://127.9.231.65:35713/ using document root <none> and password file <none>
I20260620 02:21:49.161186 15871 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:49.161330 15871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:49.161680 15871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:49.164945 15871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "ea2d7d1c0f874e43b86ccbada09f748d"
format_stamp: "Formatted at 2026-06-20 02:21:49 on dist-test-slave-9vg1"
server_key: "b2c5a7f77c370571a7feb602bc598929"
server_key_iv: "5307579ab56ffb5f7996be10582c2fdc"
server_key_version: "encryptionkey@0"
I20260620 02:21:49.165937 15871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "ea2d7d1c0f874e43b86ccbada09f748d"
format_stamp: "Formatted at 2026-06-20 02:21:49 on dist-test-slave-9vg1"
server_key: "b2c5a7f77c370571a7feb602bc598929"
server_key_iv: "5307579ab56ffb5f7996be10582c2fdc"
server_key_version: "encryptionkey@0"
I20260620 02:21:49.172070 15871 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:21:49.175931 15886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:49.177464 15871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 02:21:49.177690 15871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "ea2d7d1c0f874e43b86ccbada09f748d"
format_stamp: "Formatted at 2026-06-20 02:21:49 on dist-test-slave-9vg1"
server_key: "b2c5a7f77c370571a7feb602bc598929"
server_key_iv: "5307579ab56ffb5f7996be10582c2fdc"
server_key_version: "encryptionkey@0"
I20260620 02:21:49.177930 15871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:49.194276 15871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:49.198393 15871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:49.198716 15871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:49.199784 15871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:49.201208 15871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:49.201320 15871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:49.201483 15871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:49.201566 15871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:49.238381 15871 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:41219
I20260620 02:21:49.238415 15998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:41219 every 8 connection(s)
I20260620 02:21:49.239822 15871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:49.245762 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 15871
I20260620 02:21:49.246045 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:49.246814 10141 external_mini_cluster.cc:1428] Setting key 98ef8ddd561d2f5b8dd49c289673a303
Jun 20 02:21:49 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922109, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:49.262051 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:49.242630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:37947 (local address 127.9.231.126:40923)
0620 02:21:49.243214 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:49.243219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:49.244045 (+ 826us) server_negotiation.cc:407] Connection header received
0620 02:21:49.246108 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:49.246117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:49.246222 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:49.246363 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:49.248147 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.248975 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.250738 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.251269 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.251424 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:49.255320 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:49.255325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:49.255329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:49.255365 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:49.258546 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:49.259493 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:49.259499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:49.259503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:49.259650 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:49.260123 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:49.260128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:49.260132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:49.260428 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:49.260601 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:49.261393 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 02:21:49.261668 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
I20260620 02:21:49.263165 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:49.243128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40923 (local address 127.9.231.65:37947)
0620 02:21:49.243868 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:49.243912 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:49.245737 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:49.246590 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:49.246603 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:49.247320 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:49.247949 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:49.247967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.249181 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.249185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:49.250553 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:49.250568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.251547 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.251554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:49.251822 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:49.252580 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:49.252609 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:49.254984 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:49.258743 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:49.258753 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:49.258779 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:49.259343 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:49.259772 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:49.259778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:49.259787 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:49.259992 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:49.260581 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:49.260589 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:49.260975 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 02:21:49.261274 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:21:49.261862 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":292,"threads_started":1}
I20260620 02:21:49.265034 15999 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40923
I20260620 02:21:49.265403 15999 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:49.266288 15999 heartbeater.cc:511] Master 127.9.231.126:40923 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:21:49.268745 15808 ts_manager.cc:194] Registered new tserver with Master: ea2d7d1c0f874e43b86ccbada09f748d (127.9.231.65:41219)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:21:49.270699 15808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:37947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:21:49.299860 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40923
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:49.633769 16005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:49.634071 16005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:49.634217 16005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:49.641567 16005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:49.641685 16005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:49.641875 16005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:49.651973 16005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:49.653406 16005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:49.654810 16005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:49.668828 16011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:49.668916 16012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:49.668919 16014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:49.669981 16005 server_base.cc:1034] running on GCE node
I20260620 02:21:49.670624 16005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:49.671727 16005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:49.672947 16005 hybrid_clock.cc:648] HybridClock initialized: now 1781922109672863 us; error 73 us; skew 500 ppm
I20260620 02:21:49.673327 16005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:49 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922109, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:49.676874 16005 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:21:49.678535 16005 webserver.cc:460] Webserver started at http://127.9.231.66:37169/ using document root <none> and password file <none>
I20260620 02:21:49.679365 16005 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:49.679517 16005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:49.679853 16005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:49.683166 16005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "0f299e99425644c7a5ede095addffdb4"
format_stamp: "Formatted at 2026-06-20 02:21:49 on dist-test-slave-9vg1"
server_key: "7bc3bbde0a39b32dc600b6e01971608a"
server_key_iv: "cc2fe02571cb833120a829562020c6ad"
server_key_version: "encryptionkey@0"
I20260620 02:21:49.684024 16005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "0f299e99425644c7a5ede095addffdb4"
format_stamp: "Formatted at 2026-06-20 02:21:49 on dist-test-slave-9vg1"
server_key: "7bc3bbde0a39b32dc600b6e01971608a"
server_key_iv: "cc2fe02571cb833120a829562020c6ad"
server_key_version: "encryptionkey@0"
I20260620 02:21:49.689498 16005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:21:49.693292 16021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:49.694849 16005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:21:49.695168 16005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "0f299e99425644c7a5ede095addffdb4"
format_stamp: "Formatted at 2026-06-20 02:21:49 on dist-test-slave-9vg1"
server_key: "7bc3bbde0a39b32dc600b6e01971608a"
server_key_iv: "cc2fe02571cb833120a829562020c6ad"
server_key_version: "encryptionkey@0"
I20260620 02:21:49.695429 16005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:49.732286 16005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:49.737440 16005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:49.737802 16005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:49.738860 16005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:49.740324 16005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:49.740402 16005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:49.740545 16005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:49.740605 16005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:49.779491 16005 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:38033
I20260620 02:21:49.779537 16133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:38033 every 8 connection(s)
I20260620 02:21:49.780884 16005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:49.783257 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16005
I20260620 02:21:49.783468 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:49.783994 10141 external_mini_cluster.cc:1428] Setting key 51e991f420139907ec2a9cca335b4aa0
Jun 20 02:21:49 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922109, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:49.805969 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:49.783885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:43011 (local address 127.9.231.126:40923)
0620 02:21:49.784459 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:49.784464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:49.785540 (+ 1076us) server_negotiation.cc:407] Connection header received
0620 02:21:49.787698 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:49.787704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:49.787799 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:49.787936 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:49.789627 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.790376 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.792576 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.793208 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.793366 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:49.797613 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:49.797623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:49.797632 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:49.797686 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:49.801732 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:49.802777 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:49.802786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:49.802792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:49.802975 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:49.803623 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:49.803631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:49.803638 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:49.804109 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:49.804295 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:49.805219 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 02:21:49.805501 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":330,"threads_started":1}
I20260620 02:21:49.806852 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:49.784541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40923 (local address 127.9.231.66:43011)
0620 02:21:49.785376 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:49.785420 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:49.787375 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:49.788217 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:49.788227 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:49.788796 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:49.789427 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:49.789446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.790566 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.790570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:49.792361 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:49.792384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:49.793454 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:49.793460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:49.793703 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:49.794491 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:49.794516 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:49.797270 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:49.801958 (+ 4688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:49.801966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:49.801984 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:49.802590 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:49.803229 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:49.803233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:49.803236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:49.803466 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:49.804318 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:49.804326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:49.804751 (+ 425us) client_negotiation.cc:769] Sending connection context
0620 02:21:49.805027 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 02:21:49.805539 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":419,"threads_started":1}
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:21:49.808837 16134 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40923
I20260620 02:21:49.809319 16134 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:49.810237 16134 heartbeater.cc:511] Master 127.9.231.126:40923 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
I20260620 02:21:49.812350 15808 ts_manager.cc:194] Registered new tserver with Master: 0f299e99425644c7a5ede095addffdb4 (127.9.231.66:38033)
I20260620 02:21:49.813815 15808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:43011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:21:49.839279 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40923
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:50.201952 16140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:50.202342 16140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:50.202512 16140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:50.210464 16140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:50.210667 16140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:50.210850 16140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:50.223193 16140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:50.224745 16140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:50.226220 16140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:50.241784 16146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:50.241933 16145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:50.242195 16148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:50.242758 16140 server_base.cc:1034] running on GCE node
I20260620 02:21:50.243589 16140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:50.244801 16140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:50.246080 16140 hybrid_clock.cc:648] HybridClock initialized: now 1781922110246013 us; error 66 us; skew 500 ppm
I20260620 02:21:50.246483 16140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.255617 16140 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:21:50.257517 16140 webserver.cc:460] Webserver started at http://127.9.231.67:36621/ using document root <none> and password file <none>
I20260620 02:21:50.258391 16140 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:50.258582 16140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:50.258952 16140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:50.262892 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "0b1dafda9634442db709664d5d1b6dd2"
format_stamp: "Formatted at 2026-06-20 02:21:50 on dist-test-slave-9vg1"
server_key: "a00e9fcb4fb5b3cbf230406c69a9595c"
server_key_iv: "be8033ad2c2c729c0ff3780a8642325e"
server_key_version: "encryptionkey@0"
I20260620 02:21:50.264014 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "0b1dafda9634442db709664d5d1b6dd2"
format_stamp: "Formatted at 2026-06-20 02:21:50 on dist-test-slave-9vg1"
server_key: "a00e9fcb4fb5b3cbf230406c69a9595c"
server_key_iv: "be8033ad2c2c729c0ff3780a8642325e"
server_key_version: "encryptionkey@0"
I20260620 02:21:50.270442 16140 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:21:50.274950 16155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:50.275398 15999 heartbeater.cc:503] Master 127.9.231.126:40923 was elected leader, sending a full tablet report...
I20260620 02:21:50.277268 16140 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:21:50.277500 16140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "0b1dafda9634442db709664d5d1b6dd2"
format_stamp: "Formatted at 2026-06-20 02:21:50 on dist-test-slave-9vg1"
server_key: "a00e9fcb4fb5b3cbf230406c69a9595c"
server_key_iv: "be8033ad2c2c729c0ff3780a8642325e"
server_key_version: "encryptionkey@0"
I20260620 02:21:50.277817 16140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:50.297058 16140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:50.302759 16140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:50.303176 16140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:50.304379 16140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:50.306075 16140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:50.306201 16140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:50.306367 16140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:50.306442 16140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:50.344360 16140 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:41007
I20260620 02:21:50.344420 16268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:41007 every 8 connection(s)
I20260620 02:21:50.345794 16140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:50.353761 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16140
I20260620 02:21:50.353978 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:50.354513 10141 external_mini_cluster.cc:1428] Setting key 8a24b5e1659f99e1d81a6a4643837376
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:21:50.369051 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.348794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:41179 (local address 127.9.231.126:40923)
0620 02:21:50.349276 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.349281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.350241 (+ 960us) server_negotiation.cc:407] Connection header received
0620 02:21:50.352294 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.352300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.352440 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.352590 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.354483 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.355614 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.357423 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.357856 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.358003 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.362094 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:50.362102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:50.362107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:50.362150 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:50.365137 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.366283 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.366299 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.366302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.366467 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.366970 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.366975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.366977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.367408 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:50.367592 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:50.368392 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 02:21:50.368723 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":210,"threads_started":1}
I20260620 02:21:50.370417 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.349240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40923 (local address 127.9.231.67:41179)
0620 02:21:50.349983 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:50.350028 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:50.351935 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:50.352864 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:50.352874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:50.353482 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:50.354266 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:50.354293 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.355811 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.355815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:50.357252 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:50.357267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.358166 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.358177 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:50.358445 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.359267 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:50.359297 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:50.361767 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:50.365371 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:50.365380 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:50.365398 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:50.365912 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:50.366596 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:50.366600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:50.366602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:50.366832 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:50.367585 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:50.367592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:50.367984 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 02:21:50.368283 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:21:50.368964 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":294,"threads_started":1}
I20260620 02:21:50.372037 16269 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40923
I20260620 02:21:50.372370 16269 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:50.373159 16269 heartbeater.cc:511] Master 127.9.231.126:40923 requested a full tablet report, sending...
I20260620 02:21:50.374908 15808 ts_manager.cc:194] Registered new tserver with Master: 0b1dafda9634442db709664d5d1b6dd2 (127.9.231.67:41007)
I20260620 02:21:50.375840 15808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:41179
I20260620 02:21:50.384944 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/test-admin.keytab.
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.413318 10141 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 02:21:50.418933 16277 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:50.416328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.9.231.65:41219)
0620 02:21:50.416939 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.416946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.416974 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:21:50.417036 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.417042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.417254 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.417396 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.418417 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35934: BlockingRecv error: recv got EOF from 127.0.0.1:35934 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":305,"threads_started":1}
W20260620 02:21:50.419378 10141 init.cc:244] Error while opening credential cache '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc)
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.420994 10141 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.422696 10141 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:50.441505 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.425334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.9.231.65:41219)
0620 02:21:50.425561 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.425568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.425799 (+ 231us) server_negotiation.cc:407] Connection header received
0620 02:21:50.426004 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.426010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.426147 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.426265 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.427415 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.428540 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.429910 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.430590 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.430740 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.434964 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:50.434983 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:50.434998 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:50.435049 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:50.438353 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.439289 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.439297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.439300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.439439 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.439856 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.439860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.439863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.440254 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:50.440589 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:50.440928 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 02:21:50.441209 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:21:50.448108 16277 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:50.445958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.9.231.65:41219)
0620 02:21:50.446142 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.446147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.446561 (+ 414us) server_negotiation.cc:407] Connection header received
0620 02:21:50.446685 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.446689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.446774 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.446869 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.447814 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35952: BlockingRecv error: recv got EOF from 127.0.0.1:35952 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 02:21:50.448273 10141 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.449946 10141 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.451593 10141 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:50.469045 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.453131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.9.231.65:41219)
0620 02:21:50.453377 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.453385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.453677 (+ 292us) server_negotiation.cc:407] Connection header received
0620 02:21:50.453907 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.453912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.453976 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.454081 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.455468 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.456275 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.457666 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.458162 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.458323 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.462780 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:50.462797 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:50.462801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:50.462847 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:50.465850 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.466766 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.466771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.466774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.466941 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.467465 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.467469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.467471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.467765 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:50.467984 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:50.468453 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 02:21:50.468648 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:21:50.474875 16277 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:21:50.472698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.9.231.65:41219)
0620 02:21:50.472933 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.472941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.473194 (+ 253us) server_negotiation.cc:407] Connection header received
0620 02:21:50.473434 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.473447 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.473537 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.473658 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.474556 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35960: BlockingRecv error: recv got EOF from 127.0.0.1:35960 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 02:21:50.475049 10141 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922055280103-10141-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.476814 10141 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:50.478444 10141 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:21:50.495327 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.480586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.9.231.65:41219)
0620 02:21:50.480794 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.480800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.480822 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:21:50.480950 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.480954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.481017 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.481138 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.482335 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.483262 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.484671 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.485163 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.485316 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.489474 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:50.489480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:50.489483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:50.489525 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:50.492331 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.493124 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.493134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.493136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.493304 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.493743 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.493748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.493752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.494064 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:50.494407 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:50.494682 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 02:21:50.494914 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:21:50.497491 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15871
I20260620 02:21:50.511308 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.498551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.9.231.65:41219)
0620 02:21:50.498767 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.498772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.499183 (+ 411us) server_negotiation.cc:407] Connection header received
0620 02:21:50.499383 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.499386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.499460 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.499548 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.500875 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.501619 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.502991 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.503550 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.503668 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.505996 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:50.506001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:50.506004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:50.506039 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:50.508743 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.509414 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.509418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.509420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.509547 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.509885 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.509888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.509890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.510146 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:50.510346 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:50.510723 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 02:21:50.510898 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 02:21:50.511969 15994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:50.602434 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15871
I20260620 02:21:50.619957 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16005
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:21:50.637799 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.621258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.9.231.66:38033)
0620 02:21:50.621918 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.621925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.621951 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:21:50.622019 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.622026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.622254 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.622392 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.623533 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.624650 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.626005 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.626645 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.626756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.631400 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:50.631417 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:50.631433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:50.631523 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:50.634576 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.635624 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.635633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.635635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.635769 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.636134 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.636138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.636140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.636526 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:50.636801 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:50.637112 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 02:21:50.637379 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":322,"threads_started":1}
I20260620 02:21:50.639422 16129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:50.727010 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16005
I20260620 02:21:50.744176 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16140
Jun 20 02:21:50 dist-test-slave-9vg1 krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:21:50.762837 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:50.745462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.9.231.67:41007)
0620 02:21:50.746260 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:50.746268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:50.746302 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:21:50.746369 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:50.746377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:50.746634 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:50.746780 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:50.748230 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.749325 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.750687 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:50.751502 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:50.751623 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:50.756205 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:50.756222 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:50.756240 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:50.756283 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:50.759389 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.760264 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.760274 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.760277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.760439 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:50.760866 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:50.760869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:50.760872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:50.761414 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:50.761699 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:50.762125 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 02:21:50.762472 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":452,"threads_started":1}
I20260620 02:21:50.764479 16264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:50.851110 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16140
I20260620 02:21:50.869352 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 15776
I20260620 02:21:50.870857 15838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:50.979246 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 15776
2026-06-20T02:21:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4222 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:21:51 dist-test-slave-9vg1 krb5kdc[16298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:21:51 dist-test-slave-9vg1 krb5kdc[16298](info): set up 2 sockets
Jun 20 02:21:51 dist-test-slave-9vg1 krb5kdc[16298](info): commencing operation
krb5kdc: starting...
W20260620 02:21:52.120970 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:21:52 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:21:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab.
Entry for principal oryx/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab.
I20260620 02:21:52.363477 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:46181
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab
--principal=oryx/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:52.722647 16311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:52.723031 16311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:52.723264 16311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:52.731204 16311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:52.731361 16311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:52.731446 16311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:52.731549 16311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:52.731592 16311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:52.743045 16311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab
--principal=oryx/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:52.744554 16311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:52.746068 16311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:52.759548 16317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:52.759563 16316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:52.759922 16319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:52.761345 16311 server_base.cc:1034] running on GCE node
I20260620 02:21:52.762288 16311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:52.763949 16311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:52.765203 16311 hybrid_clock.cc:648] HybridClock initialized: now 1781922112765136 us; error 61 us; skew 500 ppm
I20260620 02:21:52.765748 16311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:52 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:52.769395 16311 init.cc:377] Logged in from keytab as oryx/127.9.231.126@KRBTEST.COM (short username oryx)
I20260620 02:21:52.771376 16311 webserver.cc:460] Webserver started at http://127.9.231.126:34801/ using document root <none> and password file <none>
I20260620 02:21:52.772167 16311 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:52.772312 16311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:52.772675 16311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:52.776320 16311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2"
format_stamp: "Formatted at 2026-06-20 02:21:52 on dist-test-slave-9vg1"
server_key: "0fcf8e1b38220f0546441f25ecf44c5d"
server_key_iv: "5b2b442981d6361e8e1c40e64d626457"
server_key_version: "encryptionkey@0"
I20260620 02:21:52.777297 16311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2"
format_stamp: "Formatted at 2026-06-20 02:21:52 on dist-test-slave-9vg1"
server_key: "0fcf8e1b38220f0546441f25ecf44c5d"
server_key_iv: "5b2b442981d6361e8e1c40e64d626457"
server_key_version: "encryptionkey@0"
I20260620 02:21:52.783651 16311 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:21:52.788304 16326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:52.790246 16311 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:21:52.790501 16311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2"
format_stamp: "Formatted at 2026-06-20 02:21:52 on dist-test-slave-9vg1"
server_key: "0fcf8e1b38220f0546441f25ecf44c5d"
server_key_iv: "5b2b442981d6361e8e1c40e64d626457"
server_key_version: "encryptionkey@0"
I20260620 02:21:52.790752 16311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:52.805784 16311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:52.811295 16311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:52.811640 16311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:52.833820 16311 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:46181
I20260620 02:21:52.833860 16377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:46181 every 8 connection(s)
I20260620 02:21:52.835628 16311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:21:52.838912 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16311
I20260620 02:21:52.839267 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:21:52.839972 10141 external_mini_cluster.cc:1428] Setting key 25e5a4311208252f6c6e350fc6de6677
I20260620 02:21:52.840147 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:52.857213 16378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: Bootstrap starting.
Jun 20 02:21:52 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:21:52.862376 16378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:52.864153 16378 log.cc:824] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:52.868594 16378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: No bootstrap required, opened a new log
I20260620 02:21:52.870518 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:52.842160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.9.231.126:46181)
0620 02:21:52.843112 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:52.843135 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:52.843335 (+ 200us) server_negotiation.cc:407] Connection header received
0620 02:21:52.845583 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:52.845617 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:52.846232 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:52.846679 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:52.848970 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:52.850572 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:52.852625 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:52.853697 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:52.853877 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:52.859431 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:52.859448 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:52.859474 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:52.859549 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:52.864939 (+ 5390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:52.866244 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:52.866255 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:52.866266 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:52.866511 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:52.867032 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:52.867038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:52.867042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:52.867720 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:52.868100 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:52.868625 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 02:21:52.869317 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":426,"threads_started":1}
I20260620 02:21:52.877172 16378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } }
I20260620 02:21:52.877696 16378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:52.877835 16378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6be40cafc84abbbaaeeecf9f4c55a2, State: Initialized, Role: FOLLOWER
I20260620 02:21:52.878448 16378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } }
I20260620 02:21:52.878720 16378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:21:52.878871 16378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:21:52.879120 16378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:52.881431 16378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } }
I20260620 02:21:52.881994 16378 leader_election.cc:302] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d6be40cafc84abbbaaeeecf9f4c55a2; no voters:
I20260620 02:21:52.882728 16378 leader_election.cc:288] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:21:52.882903 16383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:52.883847 16383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d6be40cafc84abbbaaeeecf9f4c55a2, State: Running, Role: LEADER
I20260620 02:21:52.884486 16383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } }
I20260620 02:21:52.885111 16378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:52.887832 16385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6be40cafc84abbbaaeeecf9f4c55a2. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } }
I20260620 02:21:52.888811 16385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:52.887904 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } }
I20260620 02:21:52.889776 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:52.890873 16392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:21:52.896106 16392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:21:52.906529 16392 catalog_manager.cc:1347] Generated new cluster ID: b84011e6ba2e471e92c06586c49e4a4f
I20260620 02:21:52.906689 16392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:21:52.936964 16392 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:21:52.939030 16392 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:21:52.947019 16392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: Generated new TSK 0
I20260620 02:21:52.948149 16392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab.
Entry for principal oryx/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab.
I20260620 02:21:53.016367 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:46181
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab
--principal=oryx/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:53.411406 16404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:53.411796 16404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:53.412001 16404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:53.420077 16404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:53.420295 16404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:53.420523 16404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:53.431448 16404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab
--principal=oryx/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:46181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:53.432842 16404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:53.434265 16404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:53.449707 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:53.449707 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:53.449666 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:53.450562 16404 server_base.cc:1034] running on GCE node
I20260620 02:21:53.451344 16404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:53.452420 16404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:53.453678 16404 hybrid_clock.cc:648] HybridClock initialized: now 1781922113453634 us; error 39 us; skew 500 ppm
I20260620 02:21:53.454074 16404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:53 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922113, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:53.457659 16404 init.cc:377] Logged in from keytab as oryx/127.9.231.65@KRBTEST.COM (short username oryx)
I20260620 02:21:53.459432 16404 webserver.cc:460] Webserver started at http://127.9.231.65:34041/ using document root <none> and password file <none>
I20260620 02:21:53.460187 16404 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:53.460337 16404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:53.460659 16404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:53.463843 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "bbf949289be24f38b43ca2d8ebe681d8"
format_stamp: "Formatted at 2026-06-20 02:21:53 on dist-test-slave-9vg1"
server_key: "60561cf6334386c7d799ff58d9134f15"
server_key_iv: "82a92edfa0fcec2a2d53f04044da6cfb"
server_key_version: "encryptionkey@0"
I20260620 02:21:53.464670 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "bbf949289be24f38b43ca2d8ebe681d8"
format_stamp: "Formatted at 2026-06-20 02:21:53 on dist-test-slave-9vg1"
server_key: "60561cf6334386c7d799ff58d9134f15"
server_key_iv: "82a92edfa0fcec2a2d53f04044da6cfb"
server_key_version: "encryptionkey@0"
I20260620 02:21:53.470243 16404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:21:53.474247 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:53.475924 16404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:53.476130 16404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "bbf949289be24f38b43ca2d8ebe681d8"
format_stamp: "Formatted at 2026-06-20 02:21:53 on dist-test-slave-9vg1"
server_key: "60561cf6334386c7d799ff58d9134f15"
server_key_iv: "82a92edfa0fcec2a2d53f04044da6cfb"
server_key_version: "encryptionkey@0"
I20260620 02:21:53.476370 16404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:53.508733 16404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:53.513897 16404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:53.514348 16404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:53.515954 16404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:53.517634 16404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:53.517724 16404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:53.517944 16404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:53.518002 16404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:53.558121 16404 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:41181
I20260620 02:21:53.558188 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:41181 every 8 connection(s)
I20260620 02:21:53.559666 16404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:53.563890 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16404
I20260620 02:21:53.564180 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:21:53.565061 10141 external_mini_cluster.cc:1428] Setting key 4a7c36dc1969acedfdb3d572f339653f
Jun 20 02:21:53 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922113, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:21:53.586325 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:53.562747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:37961 (local address 127.9.231.126:46181)
0620 02:21:53.563323 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:53.563328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:53.564553 (+ 1225us) server_negotiation.cc:407] Connection header received
0620 02:21:53.566907 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:53.566916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:53.567014 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:53.567181 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:53.569077 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:53.569874 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:53.571846 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:53.572375 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:53.572536 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:53.577952 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:53.577961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:53.577965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:53.578021 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:53.581712 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:53.582949 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:53.582954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:53.582956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:53.583201 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:53.583817 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:53.583821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:53.583823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:53.584148 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:53.584270 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:53.585535 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0620 02:21:53.585903 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":332,"threads_started":1}
I20260620 02:21:53.587975 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:53.563382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.65:37961)
0620 02:21:53.564355 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:53.564402 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:53.566490 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:53.567487 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:53.567499 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:53.568150 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:53.568843 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:53.568865 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:53.570100 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:53.570105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:53.571646 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:53.571663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:53.572668 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:53.572677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:53.572966 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:53.573867 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:53.573908 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:53.577591 (+ 3683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:53.581926 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:53.581940 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:53.581962 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:53.582758 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:53.583343 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:53.583351 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:53.583358 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:53.583675 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:53.584332 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:53.584344 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:53.585030 (+ 686us) client_negotiation.cc:769] Sending connection context
0620 02:21:53.585387 (+ 357us) client_negotiation.cc:240] Negotiation successful
0620 02:21:53.586099 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":371,"threads_started":1}
I20260620 02:21:53.590059 16533 heartbeater.cc:347] Connected to a master server at 127.9.231.126:46181
I20260620 02:21:53.590512 16533 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:21:53.591570 16533 heartbeater.cc:511] Master 127.9.231.126:46181 requested a full tablet report, sending...
I20260620 02:21:53.594237 16343 ts_manager.cc:194] Registered new tserver with Master: bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.66@KRBTEST.COM" created.
I20260620 02:21:53.596909 16343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:37961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab.
Entry for principal oryx/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab.
I20260620 02:21:53.626710 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:46181
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab
--principal=oryx/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:53.987231 16539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:53.987566 16539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:53.987666 16539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:53.995491 16539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:53.995662 16539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:53.995813 16539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:54.006441 16539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab
--principal=oryx/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:46181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:54.007887 16539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:54.009320 16539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:54.024573 16547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:54.024804 16545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:54.024659 16544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:54.026422 16539 server_base.cc:1034] running on GCE node
I20260620 02:21:54.027330 16539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:54.028481 16539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:54.029735 16539 hybrid_clock.cc:648] HybridClock initialized: now 1781922114029684 us; error 50 us; skew 500 ppm
I20260620 02:21:54.030165 16539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:54 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922114, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:54.033972 16539 init.cc:377] Logged in from keytab as oryx/127.9.231.66@KRBTEST.COM (short username oryx)
I20260620 02:21:54.035966 16539 webserver.cc:460] Webserver started at http://127.9.231.66:45767/ using document root <none> and password file <none>
I20260620 02:21:54.036813 16539 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:54.036986 16539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:54.037333 16539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:54.040799 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "d9732a2886db4b35ac7137c80866be27"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "1e18da581b552256d55f4bfc6711deac"
server_key_iv: "70083f939fc9828b96011f5f4e41ff1e"
server_key_version: "encryptionkey@0"
I20260620 02:21:54.041702 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "d9732a2886db4b35ac7137c80866be27"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "1e18da581b552256d55f4bfc6711deac"
server_key_iv: "70083f939fc9828b96011f5f4e41ff1e"
server_key_version: "encryptionkey@0"
I20260620 02:21:54.047981 16539 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:21:54.052150 16554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:54.053850 16539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:21:54.054127 16539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "d9732a2886db4b35ac7137c80866be27"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "1e18da581b552256d55f4bfc6711deac"
server_key_iv: "70083f939fc9828b96011f5f4e41ff1e"
server_key_version: "encryptionkey@0"
I20260620 02:21:54.054391 16539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:54.084448 16539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:54.089762 16539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:54.090096 16539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:54.091359 16539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:54.092861 16539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:54.092974 16539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:54.093132 16539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:54.093207 16539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:54.129371 16539 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:40215
I20260620 02:21:54.129418 16667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:40215 every 8 connection(s)
I20260620 02:21:54.130779 16539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:54.131489 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16539
I20260620 02:21:54.131733 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:21:54.132330 10141 external_mini_cluster.cc:1428] Setting key 3432f072317f087cff7561d64d3bf486
Jun 20 02:21:54 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922114, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:21:54.156155 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:54.133931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:53461 (local address 127.9.231.126:46181)
0620 02:21:54.134502 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:54.134508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:54.135693 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 02:21:54.137635 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:54.137644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:54.137746 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:54.137886 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:54.139636 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.140939 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.143700 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.144242 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.144363 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:54.149428 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:54.149435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:54.149438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:54.149483 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:54.152456 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:54.153401 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:54.153406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:54.153408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:54.153556 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:54.154068 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:54.154073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:54.154075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:54.154357 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:54.154522 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:54.155474 (+ 952us) server_negotiation.cc:299] Negotiation successful
0620 02:21:54.155736 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":325,"threads_started":1}
I20260620 02:21:54.157429 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:54.134623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.66:53461)
0620 02:21:54.135510 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:54.135560 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:54.137312 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:54.138116 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:54.138127 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:54.138710 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:54.139418 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:54.139439 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.141129 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.141135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:54.143500 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:54.143522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.144521 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.144531 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:54.144939 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:54.145810 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:54.145847 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:54.148935 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:54.152664 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:54.152676 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:54.152701 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:54.153234 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:54.153686 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:54.153691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:54.153695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:54.153923 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:54.154511 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:54.154521 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:54.154977 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 02:21:54.155348 (+ 371us) client_negotiation.cc:240] Negotiation successful
0620 02:21:54.156104 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":394,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.67@KRBTEST.COM" created.
I20260620 02:21:54.159566 16668 heartbeater.cc:347] Connected to a master server at 127.9.231.126:46181
I20260620 02:21:54.159919 16668 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:54.160732 16668 heartbeater.cc:511] Master 127.9.231.126:46181 requested a full tablet report, sending...
I20260620 02:21:54.162441 16343 ts_manager.cc:194] Registered new tserver with Master: d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215)
I20260620 02:21:54.163401 16343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:53461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab.
Entry for principal oryx/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab.
I20260620 02:21:54.187569 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:46181
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab
--principal=oryx/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:54.541410 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:54.541745 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:54.541914 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:54.549649 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:54.549778 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:54.549983 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:54.561237 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab
--principal=oryx/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:46181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:54.562712 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:54.564215 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:54.579288 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:54.579289 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:54.579509 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:54.580190 16674 server_base.cc:1034] running on GCE node
I20260620 02:21:54.580955 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:54.582074 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:54.583281 16674 hybrid_clock.cc:648] HybridClock initialized: now 1781922114583192 us; error 69 us; skew 500 ppm
I20260620 02:21:54.583705 16674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:54 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922114, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:54.587618 16674 init.cc:377] Logged in from keytab as oryx/127.9.231.67@KRBTEST.COM (short username oryx)
I20260620 02:21:54.589421 16674 webserver.cc:460] Webserver started at http://127.9.231.67:46033/ using document root <none> and password file <none>
I20260620 02:21:54.590313 16674 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:54.590453 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:54.590828 16674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:54.594197 16674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "3bd6534c3827497ca34cb7d4cb3cb67c"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "cb5d77a880463c61eb6aa7a8e0c5b6f4"
server_key_iv: "db3c6b72d0e71f9cdb497ea617ace896"
server_key_version: "encryptionkey@0"
I20260620 02:21:54.595341 16674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "3bd6534c3827497ca34cb7d4cb3cb67c"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "cb5d77a880463c61eb6aa7a8e0c5b6f4"
server_key_iv: "db3c6b72d0e71f9cdb497ea617ace896"
server_key_version: "encryptionkey@0"
I20260620 02:21:54.600970 16533 heartbeater.cc:503] Master 127.9.231.126:46181 was elected leader, sending a full tablet report...
I20260620 02:21:54.601773 16674 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:21:54.606537 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:54.608309 16674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:54.608530 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "3bd6534c3827497ca34cb7d4cb3cb67c"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "cb5d77a880463c61eb6aa7a8e0c5b6f4"
server_key_iv: "db3c6b72d0e71f9cdb497ea617ace896"
server_key_version: "encryptionkey@0"
I20260620 02:21:54.608745 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:54.639565 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:54.644243 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:54.644588 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:54.645705 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:54.647333 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:54.647436 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:54.647635 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:54.647708 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:54.684244 16674 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:40697
I20260620 02:21:54.684293 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:40697 every 8 connection(s)
I20260620 02:21:54.685702 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:54.692376 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16674
I20260620 02:21:54.692628 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:21:54.693220 10141 external_mini_cluster.cc:1428] Setting key e1775d82aa6c164bc1408d82caef9cde
Jun 20 02:21:54 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922114, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:21:54.709390 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:54.688719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:56393 (local address 127.9.231.126:46181)
0620 02:21:54.689191 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:54.689196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:54.690490 (+ 1294us) server_negotiation.cc:407] Connection header received
0620 02:21:54.692783 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:54.692788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:54.692881 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:54.692982 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:54.695462 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.696530 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.698415 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.698848 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.699027 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:54.702947 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:54.702953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:54.702956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:54.703024 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:54.705740 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:54.706735 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:54.706740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:54.706743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:54.706878 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:54.707439 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:54.707444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:54.707446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:54.707733 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:54.707932 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:54.708708 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 02:21:54.708960 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":208,"threads_started":1}
I20260620 02:21:54.710479 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:54.689400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:56393)
0620 02:21:54.690290 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:54.690337 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:54.692371 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:54.693351 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:54.693366 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:54.694154 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:54.695241 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:54.695273 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.696731 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.696739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:54.698230 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:54.698246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:54.699191 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:54.699201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:54.699472 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:54.700216 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:54.700241 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:54.702654 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:54.705985 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:54.705993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:54.706010 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:54.706541 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:54.707007 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:54.707012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:54.707016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:54.707301 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:54.707879 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:54.707889 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:54.708310 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 02:21:54.708603 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:21:54.709192 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":316,"threads_started":1}
I20260620 02:21:54.712070 16803 heartbeater.cc:347] Connected to a master server at 127.9.231.126:46181
I20260620 02:21:54.712399 16803 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:54.713315 16803 heartbeater.cc:511] Master 127.9.231.126:46181 requested a full tablet report, sending...
I20260620 02:21:54.715014 16343 ts_manager.cc:194] Registered new tserver with Master: 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67:40697)
I20260620 02:21:54.715909 16343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:56393
I20260620 02:21:54.722389 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.231.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.125.keytab.
Entry for principal oryx/127.9.231.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.125.keytab.
I20260620 02:21:54.780006 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:43761
--webserver_interface=127.9.231.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.125.keytab
--principal=oryx/127.9.231.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:46181,127.9.231.125:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:21:55.159178 16809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:55.159499 16809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:55.159660 16809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 02:21:55.166776 16668 heartbeater.cc:503] Master 127.9.231.126:46181 was elected leader, sending a full tablet report...
W20260620 02:21:55.167925 16809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:55.168087 16809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:55.168167 16809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:55.168239 16809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:55.168326 16809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:55.179253 16809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:46181,127.9.231.125:43761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.125.keytab
--principal=oryx/127.9.231.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.231.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
W20260620 02:21:55.179611 16809 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.231.126:46181,127.9.231.125:43761'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 02:21:55.180886 16809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:55.182315 16809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:55.195309 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:55.195587 16815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:55.195574 16817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:55.197189 16809 server_base.cc:1034] running on GCE node
I20260620 02:21:55.198154 16809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:55.199966 16809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:55.201294 16809 hybrid_clock.cc:648] HybridClock initialized: now 1781922115201228 us; error 66 us; skew 500 ppm
I20260620 02:21:55.201738 16809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:55 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922115, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:55.205588 16809 init.cc:377] Logged in from keytab as oryx/127.9.231.125@KRBTEST.COM (short username oryx)
I20260620 02:21:55.207500 16809 webserver.cc:460] Webserver started at http://127.9.231.125:46843/ using document root <none> and password file <none>
I20260620 02:21:55.208345 16809 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:55.208496 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:55.208850 16809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:55.212321 16809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/instance:
uuid: "42f9acbf4c54411481ad5350040af0f8"
format_stamp: "Formatted at 2026-06-20 02:21:55 on dist-test-slave-9vg1"
server_key: "e62beb94fab971f7897fc2cb9d8a163a"
server_key_iv: "3453aa86051e370aaa8cd2be29cd72d0"
server_key_version: "encryptionkey@0"
I20260620 02:21:55.213299 16809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal/instance:
uuid: "42f9acbf4c54411481ad5350040af0f8"
format_stamp: "Formatted at 2026-06-20 02:21:55 on dist-test-slave-9vg1"
server_key: "e62beb94fab971f7897fc2cb9d8a163a"
server_key_iv: "3453aa86051e370aaa8cd2be29cd72d0"
server_key_version: "encryptionkey@0"
I20260620 02:21:55.219677 16809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:21:55.224210 16824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:55.226155 16809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:21:55.226424 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
uuid: "42f9acbf4c54411481ad5350040af0f8"
format_stamp: "Formatted at 2026-06-20 02:21:55 on dist-test-slave-9vg1"
server_key: "e62beb94fab971f7897fc2cb9d8a163a"
server_key_iv: "3453aa86051e370aaa8cd2be29cd72d0"
server_key_version: "encryptionkey@0"
I20260620 02:21:55.226774 16809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:55.253752 16809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:55.264674 16809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:55.265051 16809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:55.286309 16809 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.125:43761
I20260620 02:21:55.286326 16875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.125:43761 every 8 connection(s)
I20260620 02:21:55.287900 16809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
I20260620 02:21:55.292260 16876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:55.295739 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16809
I20260620 02:21:55.301222 16876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:21:55 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.125@KRBTEST.COM
Jun 20 02:21:55 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922115, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.125@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:21:55.324474 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.304387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.9.231.126:46181)
0620 02:21:55.304963 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.304979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.305525 (+ 546us) server_negotiation.cc:407] Connection header received
0620 02:21:55.305764 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.305769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.305859 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.305970 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:55.307419 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.308569 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.310750 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.311455 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.311617 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.316061 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:55.316069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:55.316075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:55.316138 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:55.320027 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.320943 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.320948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.320951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.321171 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.321723 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.321728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.321732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.322333 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:55.322518 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.323744 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.324038 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
I20260620 02:21:55.325057 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.304400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:35518)
0620 02:21:55.305347 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:55.305404 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:55.305584 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:55.306077 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:55.306085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:55.306905 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:55.307243 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.307254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.308731 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.308737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.310503 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.310516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.311721 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.311726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.311859 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.312612 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:55.312659 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:55.315808 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:55.320186 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.320202 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.320216 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.320801 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.321297 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.321302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.321306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.321511 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.322499 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:55.322506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:55.322779 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 02:21:55.323237 (+ 458us) client_negotiation.cc:240] Negotiation successful
0620 02:21:55.324002 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":227,"thread_start_us":495,"threads_started":1}
I20260620 02:21:55.325671 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.298572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.9.231.125:43761)
0620 02:21:55.299489 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.299504 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.299547 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:21:55.301271 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.301295 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.301797 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.302245 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:55.304244 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.306368 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.308515 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.309702 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.309922 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.315623 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:55.315634 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:55.315649 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:55.315692 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:55.319773 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.320851 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.320897 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.320900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.321132 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.321730 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.321735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.321738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.322555 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:55.322806 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.323339 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.323714 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":447,"threads_started":1}
I20260620 02:21:55.327378 16876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:21:55 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922115, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.125@KRBTEST.COM for oryx/127.9.231.125@KRBTEST.COM
I20260620 02:21:55.341831 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.328167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.0.0.1:43802)
0620 02:21:55.328362 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:55.328378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:55.328472 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:55.328840 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:55.328844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:55.329316 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:55.329469 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.329478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.330593 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.330598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.331693 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.331707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.332443 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.332448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.332655 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.333301 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:55.333320 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:55.335764 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:55.339278 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.339284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.339289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.339706 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.340092 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.340098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.340104 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.340245 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.340812 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:55.340816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:55.340954 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 02:21:55.341097 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:21:55.341419 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 02:21:55.341832 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.328236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.9.231.125:43761)
0620 02:21:55.328395 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.328401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.328422 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:21:55.328590 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.328597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.328680 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.328801 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:55.329613 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.330435 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.331836 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.332285 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.332432 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.336002 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:55.336008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:55.336011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:55.336043 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:55.338925 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.339842 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.339850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.339854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.339978 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.340369 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.340376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.340379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.340686 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:55.340851 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.341175 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.341550 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":60}
I20260620 02:21:55.347039 16876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Bootstrap starting.
I20260620 02:21:55.350651 16876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:55.351897 16876 log.cc:824] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:55.355490 16876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: No bootstrap required, opened a new log
I20260620 02:21:55.363260 16876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } }
I20260620 02:21:55.363668 16876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:55.363866 16876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f9acbf4c54411481ad5350040af0f8, State: Initialized, Role: FOLLOWER
I20260620 02:21:55.364590 16876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } }
I20260620 02:21:55.365576 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } } }
I20260620 02:21:55.365947 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:21:55.367028 16876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:21:55.377928 16895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:21:55.378180 16895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:21:55.382613 16809 master_runner.cc:428] Detected that this master 42f9acbf4c54411481ad5350040af0f8 is joining an existing cluster
I20260620 02:21:55.382748 16809 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.231.125:43761
I20260620 02:21:55.409480 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.390951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:35528)
0620 02:21:55.391863 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:55.391885 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:55.392125 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:55.392850 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:55.392856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:55.393667 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:55.394031 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.394042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.395488 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.395497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.397377 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.397393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.398777 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.398784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.398983 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.399867 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:55.399896 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:55.401180 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:55.405878 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.405884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.405888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.406505 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.407048 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.407086 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.407089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.407280 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.408106 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:55.408111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:55.408347 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 02:21:55.408591 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 02:21:55.408939 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":126,"thread_start_us":561,"threads_started":1}
I20260620 02:21:55.410593 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.390938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.9.231.126:46181)
0620 02:21:55.391204 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.391211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.391991 (+ 780us) server_negotiation.cc:407] Connection header received
0620 02:21:55.392328 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.392339 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.392472 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.392565 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:55.394213 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.395300 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.397622 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.398331 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.398445 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.401458 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:55.401468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:55.401476 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:55.401537 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:55.405695 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.406699 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.406706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.406710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.406897 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.407438 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.407443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.407447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.407869 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:55.408039 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.409883 (+ 1844us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.410182 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:21:55.412405 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.392122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.0.0.1:43818)
0620 02:21:55.392816 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:55.392834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:55.393011 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:55.393614 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:55.393619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:55.394396 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:55.394757 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.394773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.396093 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.396098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.397315 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.397329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.398728 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.398737 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.398978 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.402726 (+ 3748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:55.402752 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:55.404008 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:55.408842 (+ 4834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.408850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.408857 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.409512 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.410000 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.410004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.410008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.410193 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.411234 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:55.411239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:55.411420 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 02:21:55.411582 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 02:21:55.411907 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":68,"thread_start_us":327,"threads_started":1}
I20260620 02:21:55.413316 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.392398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.9.231.125:43761)
0620 02:21:55.392674 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.392680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.392915 (+ 235us) server_negotiation.cc:407] Connection header received
0620 02:21:55.393158 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.393167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.393250 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.393332 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:55.394915 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.395897 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.397504 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.398263 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.398416 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.404256 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:55.404264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:55.404270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:55.404322 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:55.408684 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.409680 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.409687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.409693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.409857 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.410317 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.410324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.410329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.410741 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:55.412407 (+ 1666us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.412563 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.412852 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for oryx/127.9.231.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.124@KRBTEST.COM" created.
Jun 20 02:21:55 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.126@KRBTEST.COM for oryx/127.9.231.125@KRBTEST.COM
I20260620 02:21:55.444854 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.418474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.0.0.1:43828)
0620 02:21:55.419472 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:55.419521 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:55.419788 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:55.421340 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:55.421349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:55.422514 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:55.422882 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.422902 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.427401 (+ 4499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.427409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.429237 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.429267 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.430802 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.430812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.431097 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.432051 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:55.432086 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:55.435818 (+ 3732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:55.440769 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.440794 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.440818 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.441564 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.442243 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.442249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.442254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.442476 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.443351 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:55.443361 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:55.443616 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 02:21:55.443862 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:21:55.444355 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":482,"threads_started":1}
I20260620 02:21:55.444876 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.418539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.9.231.125:43761)
0620 02:21:55.418750 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.418756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.419620 (+ 864us) server_negotiation.cc:407] Connection header received
0620 02:21:55.420040 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.420051 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.420905 (+ 854us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.421283 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:55.423177 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.424455 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.429575 (+ 5120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.430543 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.430728 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.436168 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:55.436176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:55.436182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:55.436233 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:55.440496 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.441796 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.441806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.441812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.442051 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.442643 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.442653 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.442658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.443159 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:55.443337 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.444034 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.444387 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:21:55.447729 16343 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.231.125:43761: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "42f9acbf4c54411481ad5350040af0f8"
member_type: NON_VOTER
last_known_addr {
host: "127.9.231.125"
port: 43761
}
attrs {
promote: true
}
}
dest_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2"
cas_config_opid_index: -1
I20260620 02:21:55.449038 16343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } }
I20260620 02:21:55.452638 16908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 42f9acbf4c54411481ad5350040af0f8 (127.9.231.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } }
I20260620 02:21:55.454306 16851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:55.458264 16908 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.231.125:43761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.124.keytab.
Entry for principal oryx/127.9.231.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.124.keytab.
I20260620 02:21:55.458721 16851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Refusing update from remote peer 9d6be40cafc84abbbaaeeecf9f4c55a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:21:55.459311 16809 master.cc:537] Master@127.9.231.125:43761 shutting down...
I20260620 02:21:55.459851 16907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } }
I20260620 02:21:55.460184 16907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:55.460204 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:39611
--webserver_interface=127.9.231.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.124.keytab
--principal=oryx/127.9.231.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:21:55.462627 16907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:21:55.466825 16327 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 02:21:55.472573 16327 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 -> Peer 42f9acbf4c54411481ad5350040af0f8 (127.9.231.125:43761): Couldn't send request to peer 42f9acbf4c54411481ad5350040af0f8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:21:55.474697 16809 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:55.475250 16809 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:55.475469 16809 tablet_replica.cc:330] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: stopping tablet replica
I20260620 02:21:55.493731 16809 master.cc:552] Master@127.9.231.125:43761 shutdown complete.
I20260620 02:21:55.493840 16809 master_runner.cc:305] Clearing existing system tablet
I20260620 02:21:55.494382 16809 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:55.494488 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:55.498013 16809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 02:21:55.500028 16913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:55.500491 16809 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:55.500784 16809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 02:21:55.500878 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
uuid: "42f9acbf4c54411481ad5350040af0f8"
format_stamp: "Formatted at 2026-06-20 02:21:55 on dist-test-slave-9vg1"
server_key: "e62beb94fab971f7897fc2cb9d8a163a"
server_key_iv: "3453aa86051e370aaa8cd2be29cd72d0"
server_key_version: "encryptionkey@0"
I20260620 02:21:55.506512 16809 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:55.514210 16809 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:21:55.514459 16809 log.cc:1196] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 02:21:55.514945 16809 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Deleting consensus metadata
I20260620 02:21:55.515278 16809 master_runner.cc:315] Copying system tablet from 127.9.231.126:46181
I20260620 02:21:55.518137 16809 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: tablet copy: Beginning tablet copy session from remote peer at address 127.9.231.126:46181
I20260620 02:21:55.539909 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.519202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.9.231.126:46181)
0620 02:21:55.519408 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.519414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.523206 (+ 3792us) server_negotiation.cc:407] Connection header received
0620 02:21:55.523424 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.523430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.523544 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.523660 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:55.524926 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.526137 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.528447 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.529195 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.529349 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.532010 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:55.532018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:55.532023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:55.532072 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:55.536120 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.537314 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.537320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.537325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.537507 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:55.538044 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:55.538050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:55.538054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:55.538456 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:55.538622 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.539254 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.539517 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:21:55.539927 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.519026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:35534)
0620 02:21:55.522982 (+ 3956us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:55.523008 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:55.523274 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:55.523745 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:55.523753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:55.524318 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:55.524712 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.524726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.526360 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.526370 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.528191 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.528218 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.529408 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.529415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.529640 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.530465 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:55.530491 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:55.531690 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:55.536431 (+ 4741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.536437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.536441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.537113 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.537662 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:55.537667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:55.537670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:55.537881 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:55.538605 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:55.538610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:55.538848 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 02:21:55.539094 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:21:55.539480 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3774,"thread_start_us":3523,"threads_started":1}
I20260620 02:21:55.541359 16363 tablet_copy_service.cc:140] P 9d6be40cafc84abbbaaeeecf9f4c55a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 42f9acbf4c54411481ad5350040af0f8 ({username='oryx', principal='oryx/127.9.231.125@KRBTEST.COM'} at 127.0.0.1:35534)
I20260620 02:21:55.541684 16363 tablet_copy_service.cc:161] P 9d6be40cafc84abbbaaeeecf9f4c55a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 42f9acbf4c54411481ad5350040af0f8 at {username='oryx', principal='oryx/127.9.231.125@KRBTEST.COM'} at 127.0.0.1:35534: session id = 42f9acbf4c54411481ad5350040af0f8-00000000000000000000000000000000
I20260620 02:21:55.546587 16363 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:21:55.549456 16809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:21:55.558096 16809 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: tablet copy: Starting download of 0 data blocks...
I20260620 02:21:55.558681 16809 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: tablet copy: Starting download of 1 WAL segments...
I20260620 02:21:55.562292 16809 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:21:55.566273 16363 tablet_copy_service.cc:342] P 9d6be40cafc84abbbaaeeecf9f4c55a2: Request end of tablet copy session 42f9acbf4c54411481ad5350040af0f8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.231.125@KRBTEST.COM'} at 127.0.0.1:35534
I20260620 02:21:55.566530 16363 tablet_copy_service.cc:433] P 9d6be40cafc84abbbaaeeecf9f4c55a2: ending tablet copy session 42f9acbf4c54411481ad5350040af0f8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 42f9acbf4c54411481ad5350040af0f8
I20260620 02:21:55.578559 16809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:55.582062 16920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:55.591455 16921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:55.592984 16923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:55.596040 16809 server_base.cc:1034] running on GCE node
I20260620 02:21:55.596457 16809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:55.597325 16809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:55.598635 16809 hybrid_clock.cc:648] HybridClock initialized: now 1781922115598563 us; error 44 us; skew 500 ppm
I20260620 02:21:55.598891 16809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:55 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922115, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:55.601034 16809 init.cc:377] Logged in from keytab as oryx/127.9.231.125@KRBTEST.COM (short username oryx)
I20260620 02:21:55.607088 16809 webserver.cc:460] Webserver started at http://127.9.231.125:40355/ using document root <none> and password file <none>
I20260620 02:21:55.607508 16809 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:55.607611 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:55.610163 16809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 02:21:55.612094 16930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:55.612619 16809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 02:21:55.612787 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
uuid: "42f9acbf4c54411481ad5350040af0f8"
format_stamp: "Formatted at 2026-06-20 02:21:55 on dist-test-slave-9vg1"
server_key: "e62beb94fab971f7897fc2cb9d8a163a"
server_key_iv: "3453aa86051e370aaa8cd2be29cd72d0"
server_key_version: "encryptionkey@0"
I20260620 02:21:55.612970 16809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:55.632946 16809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:55.637063 16809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:55.664148 16809 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.125:43761
I20260620 02:21:55.664167 16981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.125:43761 every 8 connection(s)
I20260620 02:21:55.665118 16809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
I20260620 02:21:55.672420 16982 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:21:55.675108 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Bootstrap starting.
I20260620 02:21:55.691030 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:21:55.691556 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Bootstrap complete.
I20260620 02:21:55.692447 16982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } }
I20260620 02:21:55.692726 16982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42f9acbf4c54411481ad5350040af0f8, State: Initialized, Role: LEARNER
I20260620 02:21:55.692996 16982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } }
I20260620 02:21:55.693956 16985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } }
I20260620 02:21:55.694300 16982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:55.694573 16985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:21:55.708902 16996 catalog_manager.cc:1259] Loaded cluster ID: b84011e6ba2e471e92c06586c49e4a4f
I20260620 02:21:55.709069 16996 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: loading cluster ID for follower catalog manager: success
I20260620 02:21:55.711969 16996 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: acquiring CA information for follower catalog manager: success
I20260620 02:21:55.714558 16996 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:21:55.718807 16803 heartbeater.cc:503] Master 127.9.231.126:46181 was elected leader, sending a full tablet report...
I20260620 02:21:55.947664 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.940718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.0.0.1:43840)
0620 02:21:55.941012 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:55.941034 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:55.941207 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:55.942158 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:55.942163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:55.942196 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:55.942398 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.942409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.943909 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.943915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.945234 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:55.945249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.946664 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.946670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:55.946925 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.947010 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 02:21:55.947183 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 02:21:55.947344 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 02:21:55.947710 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:55.940940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.9.231.125:43761)
0620 02:21:55.941800 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:55.941804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:55.941819 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:21:55.941880 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:55.941884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:55.941951 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:55.942106 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:55.942574 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.943740 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.945426 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:55.946469 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:55.946628 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:55.946926 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:55.947222 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 02:21:55.947361 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":599,"threads_started":1}
I20260620 02:21:55.948876 16957 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 02:21:55.949584 16985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6be40cafc84abbbaaeeecf9f4c55a2. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } }
I20260620 02:21:55.949847 16985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: LEARNER
W20260620 02:21:55.972863 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:55.973198 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:55.973292 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:55.981747 16911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:21:55.981873 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:55.981921 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:55.981971 16911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:21:55.982023 16911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:21:55.993093 16911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.124.keytab
--principal=oryx/127.9.231.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.231.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:55.994592 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:55.996814 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:56.011912 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:56.011840 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:56.011873 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:56.013034 16911 server_base.cc:1034] running on GCE node
I20260620 02:21:56.014051 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:56.015779 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:56.017099 16911 hybrid_clock.cc:648] HybridClock initialized: now 1781922116017045 us; error 44 us; skew 500 ppm
I20260620 02:21:56.017529 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:56.021207 16911 init.cc:377] Logged in from keytab as oryx/127.9.231.124@KRBTEST.COM (short username oryx)
I20260620 02:21:56.022909 16911 webserver.cc:460] Webserver started at http://127.9.231.124:40999/ using document root <none> and password file <none>
I20260620 02:21:56.023793 16911 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:56.023953 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:56.024286 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:21:56.027663 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/instance:
uuid: "f6513d315fac4b87baaeb279a47978d5"
format_stamp: "Formatted at 2026-06-20 02:21:56 on dist-test-slave-9vg1"
server_key: "2ed90c342494d749a288244cca215df1"
server_key_iv: "101b13da2f99f11ee8c0ad3e9d8c7e06"
server_key_version: "encryptionkey@0"
I20260620 02:21:56.028558 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal/instance:
uuid: "f6513d315fac4b87baaeb279a47978d5"
format_stamp: "Formatted at 2026-06-20 02:21:56 on dist-test-slave-9vg1"
server_key: "2ed90c342494d749a288244cca215df1"
server_key_iv: "101b13da2f99f11ee8c0ad3e9d8c7e06"
server_key_version: "encryptionkey@0"
I20260620 02:21:56.034401 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:21:56.038475 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:56.040266 16911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:21:56.040508 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
uuid: "f6513d315fac4b87baaeb279a47978d5"
format_stamp: "Formatted at 2026-06-20 02:21:56 on dist-test-slave-9vg1"
server_key: "2ed90c342494d749a288244cca215df1"
server_key_iv: "101b13da2f99f11ee8c0ad3e9d8c7e06"
server_key_version: "encryptionkey@0"
I20260620 02:21:56.040755 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:56.061825 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:56.067018 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:56.067487 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:56.090023 16911 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.124:39611
I20260620 02:21:56.090067 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.124:39611 every 8 connection(s)
I20260620 02:21:56.091620 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
I20260620 02:21:56.096691 17065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:56.099964 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 16911
I20260620 02:21:56.106540 17065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.124@KRBTEST.COM
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.124@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:21:56.128854 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.109856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.9.231.126:46181)
0620 02:21:56.111350 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.111355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.111372 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:21:56.111447 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.111451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.111527 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.111669 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.113431 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.114585 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.116134 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.116872 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.117027 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.121705 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.121710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.121714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.121753 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.125366 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.126435 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.126443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.126446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.126646 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.127141 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.127146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.127149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.127480 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.127601 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.128260 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.128452 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"thread_start_us":1279,"threads_started":1}
I20260620 02:21:56.129281 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.109649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:35550)
0620 02:21:56.110620 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.110670 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.110853 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.111763 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.111771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.112741 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.113226 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.113240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.114795 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.114800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.115949 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.115962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.117114 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.117118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.117255 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.117958 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.117983 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.121417 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.125549 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.125566 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.125574 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.126264 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.126792 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.126797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.126800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.126976 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.127642 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.127650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.127844 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.128067 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.128416 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":478,"threads_started":1}
I20260620 02:21:56.129467 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.102796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.9.231.124:39611)
0620 02:21:56.103917 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.103931 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.103979 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:21:56.105963 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.105991 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.106466 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.106991 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.108691 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.110324 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.112227 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.113625 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.113768 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.119146 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.119160 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.119179 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.119228 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.122758 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.123832 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.123845 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.123859 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.124148 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.125021 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.125026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.125029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.125958 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.126350 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.126984 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.127651 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":501,"threads_started":1}
I20260620 02:21:56.131590 17065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.124@KRBTEST.COM for oryx/127.9.231.125@KRBTEST.COM
I20260620 02:21:56.146255 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.132687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.9.231.125:43761)
0620 02:21:56.133014 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.133020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.133038 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:21:56.133102 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.133108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.133187 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.133304 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.134040 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.134756 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.136079 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.136483 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.136637 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.140261 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.140267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.140271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.140308 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.143284 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.144158 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.144171 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.144174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.144357 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.144853 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.144856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.144859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.145207 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.145356 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.145740 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.145929 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260620 02:21:56.146291 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.132389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.0.0.1:43848)
0620 02:21:56.132639 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.132687 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.132799 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.133329 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.133334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.133727 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.133873 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.133882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.134895 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.134903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.135916 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.135929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.136660 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.136665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.136843 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.137494 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.137513 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.139895 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.143483 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.143487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.143490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.144028 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.144520 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.144523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.144526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.144709 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.145339 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.145342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.145504 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.145628 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.145945 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 02:21:56.147559 17065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.124@KRBTEST.COM for oryx/127.9.231.124@KRBTEST.COM
I20260620 02:21:56.162058 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.148287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.0.0.1:40970)
0620 02:21:56.148668 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.148706 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.148897 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.149333 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.149336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.149730 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.149913 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.149921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.150932 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.150938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.152024 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.152035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.152756 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.152762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.152915 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.153554 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.153571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.155897 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.159231 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.159235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.159238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.159684 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.160132 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.160135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.160138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.160289 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.161162 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.161165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.161315 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.161444 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.161679 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:21:56.162195 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.148918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.9.231.124:39611)
0620 02:21:56.149093 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.149098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.149113 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:21:56.149152 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.149155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.149209 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.149290 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.150064 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.150789 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.152172 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.152586 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.152690 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.156099 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.156104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.156107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.156137 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.159040 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.159796 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.159800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.159803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.159929 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.160439 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.160444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.160446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.161015 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.161175 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.161564 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.161870 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":67}
I20260620 02:21:56.167514 17065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Bootstrap starting.
I20260620 02:21:56.171259 17065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:56.172415 17065 log.cc:824] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:56.175526 17065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: No bootstrap required, opened a new log
I20260620 02:21:56.183620 17065 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } }
I20260620 02:21:56.184087 17065 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:56.184216 17065 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6513d315fac4b87baaeb279a47978d5, State: Initialized, Role: FOLLOWER
I20260620 02:21:56.184916 17065 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } }
I20260620 02:21:56.185876 17072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } } }
I20260620 02:21:56.186230 17072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:21:56.187021 17065 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:21:56.197844 17083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:21:56.198001 17083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:21:56.203295 16911 master_runner.cc:428] Detected that this master f6513d315fac4b87baaeb279a47978d5 is joining an existing cluster
I20260620 02:21:56.203473 16911 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.231.124:39611
I20260620 02:21:56.232015 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.210336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.9.231.125:43761)
0620 02:21:56.210536 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.210543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.210695 (+ 152us) server_negotiation.cc:407] Connection header received
0620 02:21:56.210975 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.210982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.211161 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.211260 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.214652 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.215922 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.218042 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.218779 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.218895 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.223541 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.223549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.223554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.223605 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.227906 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.229339 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.229346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.229349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.229511 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.230037 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.230046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.230051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.230433 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.230579 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.231317 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.231596 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:21:56.232169 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.210949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.0.0.1:40976)
0620 02:21:56.213273 (+ 2324us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.213350 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.213521 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.213938 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.213943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.214794 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.215266 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.215281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.216894 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.216901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.218719 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.218737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.220061 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.220070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.220330 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.220994 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.221048 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.223268 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.228291 (+ 5023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.228305 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.228311 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.229157 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.229674 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.229678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.229682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.229867 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.230671 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.230676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.230897 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.231173 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.231551 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2191,"mutex_wait_us":25,"spinlock_wait_cycles":25344,"thread_start_us":1936,"threads_started":1}
I20260620 02:21:56.232813 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.208511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.9.231.126:46181)
0620 02:21:56.208741 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.208748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.209695 (+ 947us) server_negotiation.cc:407] Connection header received
0620 02:21:56.209917 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.209923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.210034 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.210158 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.213937 (+ 3779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.215263 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.218609 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.219307 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.219421 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.223458 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.223468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.223473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.223535 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.228375 (+ 4840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.229674 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.229683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.229688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.229878 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.230434 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.230439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.230443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.230836 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.230974 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.231671 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.231954 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:21:56.233433 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.211422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.9.231.124:39611)
0620 02:21:56.211640 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.211645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.213440 (+ 1795us) server_negotiation.cc:407] Connection header received
0620 02:21:56.213704 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.213709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.213792 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.213923 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.215441 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.216656 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.218890 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.219775 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.220016 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.223488 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.223495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.223499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.223537 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.228065 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.229318 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.229324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.229328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.229512 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.229993 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.230002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.230009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.230436 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.230598 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.232576 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.232874 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":65}
I20260620 02:21:56.236726 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.208518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:35552)
0620 02:21:56.209516 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.209541 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.209753 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.210372 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.210377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.211661 (+ 1284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.213694 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.213722 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.215511 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.215530 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.218357 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.218373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.219825 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.219831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.220038 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.221294 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.221324 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.223157 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.228614 (+ 5457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.228619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.228623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.229467 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.230041 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.230046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.230050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.230284 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.231108 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.231113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.231344 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.233331 (+ 1987us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.236163 (+ 2832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":29,"thread_start_us":427,"threads_started":1}
I20260620 02:21:56.237339 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16404
I20260620 02:21:56.242800 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.209821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.0.0.1:43862)
0620 02:21:56.210559 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.210582 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.210815 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.211590 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.211599 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.213984 (+ 2385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.214421 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.214436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.216085 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.216092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.217822 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.217838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.219284 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.219291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.220475 (+ 1184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.221207 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.221232 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.223272 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.228140 (+ 4868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.228148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.228155 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.229157 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.229703 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.229707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.229710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.229899 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.230685 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.230688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.230918 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.232357 (+ 1439us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.240466 (+ 8109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":33,"spinlock_wait_cycles":53504,"thread_start_us":706,"threads_started":1}
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.126@KRBTEST.COM for oryx/127.9.231.124@KRBTEST.COM
I20260620 02:21:56.263964 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.243961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.9.231.124:39611)
0620 02:21:56.244136 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.244143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.244164 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:21:56.244435 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.244442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.244574 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.245797 (+ 1223us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.245998 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.246792 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.248672 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.250070 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.251261 (+ 1191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.255725 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.255735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.255740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.255793 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.260184 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.261219 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.261225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.261229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.261420 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.261920 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.261925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.261929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.262354 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.262523 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.263235 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.263548 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:21:56.263970 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.243860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.0.0.1:40982)
0620 02:21:56.244045 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.244070 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.244209 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.244786 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.244791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.245340 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.245603 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.245619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.246991 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.246996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.248457 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.248478 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.250369 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.250378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.250576 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.251543 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.251580 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.255393 (+ 3813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.260403 (+ 5010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.260410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.260414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.261039 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.261592 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.261598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.261602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.261780 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.262608 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.262613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.262861 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.263098 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.263515 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 02:21:56.266294 16343 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.231.124:39611: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f6513d315fac4b87baaeb279a47978d5"
member_type: NON_VOTER
last_known_addr {
host: "127.9.231.124"
port: 39611
}
attrs {
promote: true
}
}
dest_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2"
cas_config_opid_index: 5
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.65@KRBTEST.COM
I20260620 02:21:56.267656 16343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } }
I20260620 02:21:56.269994 16907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f6513d315fac4b87baaeb279a47978d5 (127.9.231.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.275532 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.239254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.9.231.65:41181)
0620 02:21:56.255439 (+ 16185us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.255448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.255482 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:21:56.255562 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.255574 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.255850 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.256047 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.257755 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.259283 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.261442 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.262297 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.262452 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.268130 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.268148 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.268163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.268229 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.272011 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.272876 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.272885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.272888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.273049 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.273519 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.273530 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.273532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.273905 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.274236 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.274723 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.275051 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15670,"thread_start_us":15833,"threads_started":1}
I20260620 02:21:56.276963 16528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:56.285611 17040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:56.288290 16907 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.231.124:39611
I20260620 02:21:56.289040 16957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 LEARNER]: Refusing update from remote peer 9d6be40cafc84abbbaaeeecf9f4c55a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:21:56.290045 17093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:21:56.290607 17096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.291544 17096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:56.304849 16957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f6513d315fac4b87baaeb279a47978d5 (127.9.231.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.310171 16985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: NON_VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: true } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.310513 16985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:21:56.311810 17096 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: attempting to promote NON_VOTER 42f9acbf4c54411481ad5350040af0f8 to VOTER
I20260620 02:21:56.312747 17096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } }
I20260620 02:21:56.315755 16957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 LEARNER]: Refusing update from remote peer 9d6be40cafc84abbbaaeeecf9f4c55a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 02:21:56.316565 17096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:21:56.319362 17093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 42f9acbf4c54411481ad5350040af0f8 (127.9.231.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.323048 16957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 42f9acbf4c54411481ad5350040af0f8 (127.9.231.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.324759 17093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.325122 17093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:56.325461 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.325839 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:56.327450 16911 master.cc:537] Master@127.9.231.124:39611 shutting down...
W20260620 02:21:56.327606 17018 tablet.cc:2369] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:56.328446 17040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Refusing update from remote peer 9d6be40cafc84abbbaaeeecf9f4c55a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:21:56.330227 17039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Refusing update from remote peer 9d6be40cafc84abbbaaeeecf9f4c55a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 02:21:56.331353 16985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.331630 16985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:21:56.331946 17093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:21:56.336434 16327 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 -> Peer f6513d315fac4b87baaeb279a47978d5 (127.9.231.124:39611): Couldn't send request to peer f6513d315fac4b87baaeb279a47978d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:21:56.341470 16911 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:56.341948 16911 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:56.342168 16911 tablet_replica.cc:330] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: stopping tablet replica
I20260620 02:21:56.349419 16911 master.cc:552] Master@127.9.231.124:39611 shutdown complete.
I20260620 02:21:56.349606 16911 master_runner.cc:305] Clearing existing system tablet
I20260620 02:21:56.350082 16911 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:56.350291 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:56.353646 16911 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 02:21:56.358368 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:56.358901 16911 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:56.359494 16911 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:21:56.359668 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
uuid: "f6513d315fac4b87baaeb279a47978d5"
format_stamp: "Formatted at 2026-06-20 02:21:56 on dist-test-slave-9vg1"
server_key: "2ed90c342494d749a288244cca215df1"
server_key_iv: "101b13da2f99f11ee8c0ad3e9d8c7e06"
server_key_version: "encryptionkey@0"
I20260620 02:21:56.365420 16911 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:56.374485 16911 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:21:56.374850 16911 log.cc:1196] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 02:21:56.375622 16911 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Deleting consensus metadata
I20260620 02:21:56.376004 16911 master_runner.cc:315] Copying system tablet from 127.9.231.126:46181
I20260620 02:21:56.379628 16911 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: tablet copy: Beginning tablet copy session from remote peer at address 127.9.231.126:46181
I20260620 02:21:56.405089 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.380756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.9.231.126:46181)
0620 02:21:56.381009 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.381015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.387968 (+ 6953us) server_negotiation.cc:407] Connection header received
0620 02:21:56.388216 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.388223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.388337 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.388463 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:56.390166 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.391484 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.393717 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.394495 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.394649 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.397380 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:56.397387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:56.397391 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:56.397435 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:56.401400 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.402495 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.402502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.402506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.402741 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:56.403261 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:56.403267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:56.403271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:56.403666 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:56.403875 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.404347 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.404604 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 02:21:56.405604 17107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.380716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:35558)
0620 02:21:56.387763 (+ 7047us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.387794 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.388045 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.388526 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.388531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.389477 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:56.389931 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.389950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.391726 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.391733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.393467 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.393490 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.394800 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.394809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.395028 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.395911 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:56.395941 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:56.397092 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:56.401628 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.401635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.401640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.402321 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.402877 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:56.402882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:56.402887 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:56.403122 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:56.403826 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:56.403832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:56.404044 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.404205 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.404670 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6860,"thread_start_us":6642,"threads_started":1}
I20260620 02:21:56.406312 16363 tablet_copy_service.cc:140] P 9d6be40cafc84abbbaaeeecf9f4c55a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f6513d315fac4b87baaeb279a47978d5 ({username='oryx', principal='oryx/127.9.231.124@KRBTEST.COM'} at 127.0.0.1:35558)
I20260620 02:21:56.406489 16363 tablet_copy_service.cc:161] P 9d6be40cafc84abbbaaeeecf9f4c55a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f6513d315fac4b87baaeb279a47978d5 at {username='oryx', principal='oryx/127.9.231.124@KRBTEST.COM'} at 127.0.0.1:35558: session id = f6513d315fac4b87baaeb279a47978d5-00000000000000000000000000000000
I20260620 02:21:56.410504 16363 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:21:56.412850 16911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:21:56.416882 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16404
I20260620 02:21:56.421180 16911 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: tablet copy: Starting download of 0 data blocks...
I20260620 02:21:56.421550 16911 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: tablet copy: Starting download of 1 WAL segments...
I20260620 02:21:56.424589 16911 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:21:56.431555 16363 tablet_copy_service.cc:342] P 9d6be40cafc84abbbaaeeecf9f4c55a2: Request end of tablet copy session f6513d315fac4b87baaeb279a47978d5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.231.124@KRBTEST.COM'} at 127.0.0.1:35558
I20260620 02:21:56.431748 16363 tablet_copy_service.cc:433] P 9d6be40cafc84abbbaaeeecf9f4c55a2: ending tablet copy session f6513d315fac4b87baaeb279a47978d5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f6513d315fac4b87baaeb279a47978d5
I20260620 02:21:56.442817 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:41181
--local_ip_for_outbound_sockets=127.9.231.65
--tserver_master_addrs=127.9.231.126:46181
--webserver_port=34041
--webserver_interface=127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab
--principal=oryx/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:21:56.446767 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:56.450685 17108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:56.450678 17109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:56.453809 17111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:56.454006 16911 server_base.cc:1034] running on GCE node
I20260620 02:21:56.454566 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:56.455346 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:56.456583 16911 hybrid_clock.cc:648] HybridClock initialized: now 1781922116456537 us; error 41 us; skew 500 ppm
I20260620 02:21:56.456853 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:56.459230 16911 init.cc:377] Logged in from keytab as oryx/127.9.231.124@KRBTEST.COM (short username oryx)
I20260620 02:21:56.464190 16911 webserver.cc:460] Webserver started at http://127.9.231.124:44277/ using document root <none> and password file <none>
I20260620 02:21:56.464561 16911 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:56.464708 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:56.467540 16911 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 02:21:56.469385 17119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:56.469888 16911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 02:21:56.470037 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
uuid: "f6513d315fac4b87baaeb279a47978d5"
format_stamp: "Formatted at 2026-06-20 02:21:56 on dist-test-slave-9vg1"
server_key: "2ed90c342494d749a288244cca215df1"
server_key_iv: "101b13da2f99f11ee8c0ad3e9d8c7e06"
server_key_version: "encryptionkey@0"
I20260620 02:21:56.470247 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:56.481053 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:56.484819 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:56.512239 16911 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.124:39611
I20260620 02:21:56.512259 17171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.124:39611 every 8 connection(s)
I20260620 02:21:56.513412 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
I20260620 02:21:56.521885 17172 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:21:56.528260 17172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Bootstrap starting.
I20260620 02:21:56.549201 17172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:21:56.549764 17172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: Bootstrap complete.
I20260620 02:21:56.550869 17172 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } }
I20260620 02:21:56.551249 17172 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6513d315fac4b87baaeb279a47978d5, State: Initialized, Role: LEARNER
I20260620 02:21:56.551517 17172 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } }
I20260620 02:21:56.558430 17175 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.559093 17172 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:21:56.560334 17175 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:21:56.567632 17186 catalog_manager.cc:1259] Loaded cluster ID: b84011e6ba2e471e92c06586c49e4a4f
I20260620 02:21:56.567916 17186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: loading cluster ID for follower catalog manager: success
I20260620 02:21:56.572181 17186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: acquiring CA information for follower catalog manager: success
I20260620 02:21:56.575013 17186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:21:56.874153 17113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:56.874573 17113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:56.874758 17113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:56.882994 17113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:56.883261 17113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:56.883448 17113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:21:56.895534 17113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab
--principal=oryx/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:41181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=34041
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:56.897212 17113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:56.898801 17113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:56.915231 17192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:56.915244 17193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:56.916391 17195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:56.917258 17113 server_base.cc:1034] running on GCE node
I20260620 02:21:56.918059 17113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:56.919291 17113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:56.920630 17113 hybrid_clock.cc:648] HybridClock initialized: now 1781922116920572 us; error 38 us; skew 500 ppm
I20260620 02:21:56.921039 17113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:21:56.921285 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.912792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.0.0.1:40996)
0620 02:21:56.914278 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:56.914303 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:56.914508 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:56.915182 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:56.915191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:56.915227 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:56.915451 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.915477 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.917560 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.917566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.918867 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:56.918885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.920349 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.920355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:56.920577 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.920620 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:21:56.920753 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 02:21:56.920926 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"thread_start_us":1054,"threads_started":1}
I20260620 02:21:56.921295 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:56.912902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.9.231.124:39611)
0620 02:21:56.913742 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:56.913753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:56.914636 (+ 883us) server_negotiation.cc:407] Connection header received
0620 02:21:56.914728 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:56.914734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:56.914832 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:56.914950 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:56.915885 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.917379 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.919088 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:56.920127 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:56.920297 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:56.920537 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:56.920781 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 02:21:56.920929 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":423,"threads_started":1}
I20260620 02:21:56.922398 17146 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260620 02:21:56.923032 17175 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6be40cafc84abbbaaeeecf9f4c55a2. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:56.923350 17175 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: This master's current role is: LEARNER
Jun 20 02:21:56 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:56.925622 17113 init.cc:377] Logged in from keytab as oryx/127.9.231.65@KRBTEST.COM (short username oryx)
I20260620 02:21:56.927666 17113 webserver.cc:460] Webserver started at http://127.9.231.65:34041/ using document root <none> and password file <none>
I20260620 02:21:56.928459 17113 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:56.928633 17113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:56.936055 17113 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:21:56.940129 17204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:56.941982 17113 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:21:56.942185 17113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "bbf949289be24f38b43ca2d8ebe681d8"
format_stamp: "Formatted at 2026-06-20 02:21:53 on dist-test-slave-9vg1"
server_key: "60561cf6334386c7d799ff58d9134f15"
server_key_iv: "82a92edfa0fcec2a2d53f04044da6cfb"
server_key_version: "encryptionkey@0"
I20260620 02:21:56.943437 17113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:56.985199 17113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:56.990932 17113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:56.991255 17113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:56.992360 17113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:56.994230 17113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:56.994410 17113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:56.994601 17113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:56.994675 17113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:57.035905 17113 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:41181
I20260620 02:21:57.036010 17316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:41181 every 8 connection(s)
I20260620 02:21:57.037686 17113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:21:57.038579 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 17113
I20260620 02:21:57.038982 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16539
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.124@KRBTEST.COM
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.66@KRBTEST.COM
I20260620 02:21:57.084491 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.041868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.65:39037)
0620 02:21:57.053103 (+ 11235us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:57.053176 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:57.055526 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:57.057007 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:57.057022 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:57.057814 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:57.058834 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.058860 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.060668 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.060674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.062806 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.062826 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.064138 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.064155 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.064590 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.067894 (+ 3304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:57.067955 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:57.072055 (+ 4100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:57.077765 (+ 5710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.077781 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.077814 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.078769 (+ 955us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.079470 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.079478 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.079483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.079909 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.081022 (+ 1113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:57.081033 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:57.081632 (+ 599us) client_negotiation.cc:769] Sending connection context
0620 02:21:57.082027 (+ 395us) client_negotiation.cc:240] Negotiation successful
0620 02:21:57.082697 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10800,"thread_start_us":13089,"threads_started":1}
I20260620 02:21:57.085903 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.040980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:39037 (local address 127.9.231.126:46181)
0620 02:21:57.056288 (+ 15308us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.056296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.056318 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:21:57.056379 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.056383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.056463 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.059087 (+ 2624us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.059308 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.060447 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.063115 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.063773 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.063939 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.072376 (+ 8437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.072383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.072387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.072433 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.077499 (+ 5066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.078977 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.078984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.078990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.079292 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.080154 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.080160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.080164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.080644 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.080870 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.085159 (+ 4289us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.085432 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15130,"thread_start_us":15600,"threads_started":1}
I20260620 02:21:57.089437 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.040702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.9.231.66:40215)
0620 02:21:57.053112 (+ 12410us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.053122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.053162 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:21:57.053268 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.053277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.053551 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.053775 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.056304 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.064723 (+ 8419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.072385 (+ 7662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.073396 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.073531 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.081647 (+ 8116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.081675 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.081693 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.081743 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.086062 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.087090 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.087101 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.087104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.087259 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.087669 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.087675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.087678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.088023 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.088347 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.088745 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.089029 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12222,"thread_start_us":13257,"threads_started":1}
I20260620 02:21:57.092016 16663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:57.092824 17317 heartbeater.cc:347] Connected to a master server at 127.9.231.126:46181
I20260620 02:21:57.093274 17317 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:57.094141 17317 heartbeater.cc:511] Master 127.9.231.126:46181 requested a full tablet report, sending...
I20260620 02:21:57.096522 16343 ts_manager.cc:194] Re-registered known tserver with Master: bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
I20260620 02:21:57.097728 16343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:39037
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922116, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.125@KRBTEST.COM
I20260620 02:21:57.098887 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.063332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:59813 (local address 127.9.231.124:39611)
0620 02:21:57.063644 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.063650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.065641 (+ 1991us) server_negotiation.cc:407] Connection header received
0620 02:21:57.065943 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.065949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.066049 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.066172 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.068336 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.069874 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.074744 (+ 4870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.075488 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.075672 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.080794 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.080805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.080812 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.080874 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.091008 (+ 10134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.096126 (+ 5118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.096135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.096138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.096351 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.096851 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.096856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.096859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.097339 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.097487 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.098021 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.098457 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260620 02:21:57.099850 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.056745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.9.231.65:59813)
0620 02:21:57.065539 (+ 8794us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:57.065559 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:57.065783 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:57.066197 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:57.066202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:57.067494 (+ 1292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:57.068018 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.068037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.073427 (+ 5390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.073436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.074543 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.074562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.075947 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.075956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.076167 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.076858 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:57.076887 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:57.080485 (+ 3598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:57.091955 (+ 11470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.091961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.091964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.095855 (+ 3891us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.096514 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.096520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.096524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.096710 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.097572 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:57.097576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:57.097738 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 02:21:57.099180 (+ 1442us) client_negotiation.cc:240] Negotiation successful
0620 02:21:57.099487 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8575,"thread_start_us":8684,"threads_started":1}
I20260620 02:21:57.103246 17318 heartbeater.cc:347] Connected to a master server at 127.9.231.124:39611
I20260620 02:21:57.103477 17318 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:57.104051 17318 heartbeater.cc:511] Master 127.9.231.124:39611 requested a full tablet report, sending...
I20260620 02:21:57.106355 17136 ts_manager.cc:194] Registered new tserver with Master: bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
I20260620 02:21:57.112828 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.066580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:58211 (local address 127.9.231.125:43761)
0620 02:21:57.085451 (+ 18871us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.085459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.085484 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:21:57.085553 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.085559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.085675 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.087088 (+ 1413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.087237 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.088281 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.090162 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.090826 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.091033 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.103580 (+ 12547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.103591 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.103605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.103679 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.109450 (+ 5771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.110204 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.110212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.110215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.110359 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.110891 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.110896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.110899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.111272 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.111429 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.112084 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.112326 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18703,"thread_start_us":19198,"threads_started":1}
I20260620 02:21:57.113013 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.066683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.9.231.65:58211)
0620 02:21:57.071938 (+ 5255us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:57.071970 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:57.072158 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:57.085969 (+ 13811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:57.085975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:57.086646 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:57.086918 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.086930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.088496 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.088504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.089992 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.090011 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.091109 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.091116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.091321 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.094358 (+ 3037us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x60c000034ab8. stack: 00007fdf46cb666e 00007fdf46cb6430 00007fdf45d3d5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fdf46cac0fd 00007fdf46cac977 00007fdf4d7c369d 00007fdf4d6c2810 00007fdf4d6bf470 00007fdf4d6ba33a 00007fdf4d741dca 00007fdf4d73ff68 00007fdf4d77382f 00007fdf4d77358d
0620 02:21:57.094394 (+ 36us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:57.094431 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:57.103334 (+ 8903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:57.109632 (+ 6298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.109641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.109645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.110056 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.110496 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.110502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.110506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.110744 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.111406 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:57.111411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:57.111568 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 02:21:57.111953 (+ 385us) client_negotiation.cc:240] Negotiation successful
0620 02:21:57.112326 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5014,"spinlock_wait_cycles":2478336,"thread_start_us":5573,"threads_started":1}
I20260620 02:21:57.114102 17319 heartbeater.cc:347] Connected to a master server at 127.9.231.125:43761
I20260620 02:21:57.114315 17319 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:57.114751 17319 heartbeater.cc:511] Master 127.9.231.125:43761 requested a full tablet report, sending...
I20260620 02:21:57.116587 16947 ts_manager.cc:194] Registered new tserver with Master: bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
I20260620 02:21:57.198272 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16539
I20260620 02:21:57.216282 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:40215
--local_ip_for_outbound_sockets=127.9.231.66
--tserver_master_addrs=127.9.231.126:46181
--webserver_port=45767
--webserver_interface=127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab
--principal=oryx/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:21:57.493743 17187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: NON_VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: true } } }
I20260620 02:21:57.494262 17187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:57.495697 17187 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: attempting to promote NON_VOTER f6513d315fac4b87baaeb279a47978d5 to VOTER
I20260620 02:21:57.496479 17187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } }
I20260620 02:21:57.498662 16957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Refusing update from remote peer 9d6be40cafc84abbbaaeeecf9f4c55a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:21:57.498796 17146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 LEARNER]: Refusing update from remote peer 9d6be40cafc84abbbaaeeecf9f4c55a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:21:57.499372 17187 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:21:57.499949 17331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:21:57.503209 17187 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6513d315fac4b87baaeb279a47978d5 (127.9.231.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.506078 17187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.506170 17330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.506542 17330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:57.506439 17187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:57.507268 17330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.507550 17330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:21:57.507382 16957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6513d315fac4b87baaeb279a47978d5 (127.9.231.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.511116 17336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.511739 17336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:21:57.511351 17146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6513d315fac4b87baaeb279a47978d5 (127.9.231.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.523406 17337 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:21:57.524395 17337 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:21:57.585798 17329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:57.586220 17329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:57.586386 17329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:57.594909 17329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:57.595126 17329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:57.595372 17329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:21:57.606463 17329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab
--principal=oryx/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:40215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=45767
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:57.608000 17329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:57.609541 17329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:57.624881 17343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:57.624845 17344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:57.624933 17346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:57.628923 17329 server_base.cc:1034] running on GCE node
I20260620 02:21:57.629673 17329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:57.630908 17329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:57.632191 17329 hybrid_clock.cc:648] HybridClock initialized: now 1781922117632146 us; error 74 us; skew 500 ppm
I20260620 02:21:57.632514 17329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922117, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:57.636314 17329 init.cc:377] Logged in from keytab as oryx/127.9.231.66@KRBTEST.COM (short username oryx)
I20260620 02:21:57.638479 17329 webserver.cc:460] Webserver started at http://127.9.231.66:45767/ using document root <none> and password file <none>
I20260620 02:21:57.639665 17329 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:57.639855 17329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:57.646740 17329 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:21:57.650714 17353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:57.652722 17329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:21:57.652983 17329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "d9732a2886db4b35ac7137c80866be27"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "1e18da581b552256d55f4bfc6711deac"
server_key_iv: "70083f939fc9828b96011f5f4e41ff1e"
server_key_version: "encryptionkey@0"
I20260620 02:21:57.654162 17329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:57.681857 17329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:57.688817 17329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:57.689168 17329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:57.690351 17329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:57.692108 17329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:57.692261 17329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:57.692394 17329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:57.692482 17329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:57.728964 17329 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:40215
I20260620 02:21:57.729028 17466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:40215 every 8 connection(s)
I20260620 02:21:57.730551 17329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:21:57.731130 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 17329
I20260620 02:21:57.731448 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16674
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922117, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.67@KRBTEST.COM
I20260620 02:21:57.760689 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.734012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46703 (local address 127.9.231.126:46181)
0620 02:21:57.734609 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.734615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.735941 (+ 1326us) server_negotiation.cc:407] Connection header received
0620 02:21:57.740090 (+ 4149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.740097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.740227 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.740365 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.743964 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.745285 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.747077 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.747845 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.747981 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.752203 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.752210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.752217 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.752283 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.756543 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.757431 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.757437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.757441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.757700 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.758188 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.758196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.758200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.758754 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.758895 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.759850 (+ 955us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.760141 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":292,"threads_started":1}
I20260620 02:21:57.761786 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.734673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.66:46703)
0620 02:21:57.735704 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:57.735791 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:57.739650 (+ 3859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:57.741363 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:57.741382 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:57.742484 (+ 1102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:57.743696 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.743733 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.745466 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.745491 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.746915 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.746929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.748101 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.748120 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.748445 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.749273 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:57.749309 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:57.751896 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:57.756707 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.756718 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.756744 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.757282 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.757814 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.757819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.757825 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.758055 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.758868 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:57.758876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:57.759351 (+ 475us) client_negotiation.cc:769] Sending connection context
0620 02:21:57.759680 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 02:21:57.760274 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":423,"threads_started":1}
I20260620 02:21:57.763633 17467 heartbeater.cc:347] Connected to a master server at 127.9.231.126:46181
I20260620 02:21:57.763949 17467 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:57.765064 17467 heartbeater.cc:511] Master 127.9.231.126:46181 requested a full tablet report, sending...
I20260620 02:21:57.766973 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.732686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.9.231.67:40697)
0620 02:21:57.735646 (+ 2960us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.735655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.736471 (+ 816us) server_negotiation.cc:407] Connection header received
0620 02:21:57.736559 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.736568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.736857 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.737021 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.739776 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.742444 (+ 2668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.747189 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.748234 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.749061 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.757052 (+ 7991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.757078 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.757105 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.757210 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.762697 (+ 5487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.763674 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.763682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.763686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.763851 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.764353 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.764358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.764361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.764803 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.766112 (+ 1309us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.766273 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.766514 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2782,"thread_start_us":2624,"threads_started":1}
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922117, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.124@KRBTEST.COM
I20260620 02:21:57.767279 16343 ts_manager.cc:194] Re-registered known tserver with Master: d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215)
I20260620 02:21:57.768661 16343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:46703
I20260620 02:21:57.769238 16798 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 02:21:57 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922117, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.125@KRBTEST.COM
I20260620 02:21:57.780921 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.736775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46443 (local address 127.9.231.124:39611)
0620 02:21:57.747593 (+ 10818us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.747599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.750477 (+ 2878us) server_negotiation.cc:407] Connection header received
0620 02:21:57.750767 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.750776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.750874 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.750977 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.753727 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.755030 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.758687 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.762739 (+ 4052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.762877 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.771595 (+ 8718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.771605 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.771611 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.771664 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.776920 (+ 5256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.777967 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.777975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.777980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.778168 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.778705 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.778712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.778718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.779141 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.779292 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.780260 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.780516 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10680,"thread_start_us":10528,"threads_started":1}
I20260620 02:21:57.781002 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.736361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.9.231.66:46443)
0620 02:21:57.749024 (+ 12663us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:57.749050 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:57.750601 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:57.752172 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:57.752177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:57.753045 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:57.753528 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.753545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.755248 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.755255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.758485 (+ 3230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.758499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.763204 (+ 4705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.763209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.763344 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.764940 (+ 1596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:57.764964 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:57.771289 (+ 6325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:57.777094 (+ 5805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.777100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.777104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.777780 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.778316 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.778321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.778325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.778546 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.779289 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:57.779295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:57.779530 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 02:21:57.779707 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 02:21:57.780188 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12460,"thread_start_us":11372,"threads_started":1}
I20260620 02:21:57.782091 17468 heartbeater.cc:347] Connected to a master server at 127.9.231.124:39611
I20260620 02:21:57.782294 17468 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:57.782836 17468 heartbeater.cc:511] Master 127.9.231.124:39611 requested a full tablet report, sending...
I20260620 02:21:57.783419 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.748426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:42013 (local address 127.9.231.125:43761)
0620 02:21:57.755137 (+ 6711us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:57.755144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:57.761821 (+ 6677us) server_negotiation.cc:407] Connection header received
0620 02:21:57.762734 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:57.762742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:57.762837 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:57.762914 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:57.766720 (+ 3806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.768135 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.770118 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.770756 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.770897 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.775693 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:57.775704 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:57.775709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:57.775783 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:57.779612 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.780415 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.780423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.780429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.780568 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:57.781529 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:57.781535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:57.781538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:57.781935 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:57.782125 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:57.782786 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 02:21:57.783027 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6514,"thread_start_us":302,"threads_started":1}
I20260620 02:21:57.783994 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:57.748426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.9.231.66:42013)
0620 02:21:57.761707 (+ 13281us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:57.761730 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:57.762560 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:57.763710 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:57.763716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:57.764373 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:57.764605 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.764615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.768371 (+ 3756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.768376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.769924 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:57.769943 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:57.771207 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:57.771215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:57.771462 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:57.772208 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:57.772241 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:57.775374 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:57.779752 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.779759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.779763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.780263 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.780732 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:57.780739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:57.780744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:57.780922 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:57.782266 (+ 1344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:57.782272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:57.782460 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 02:21:57.783204 (+ 744us) client_negotiation.cc:240] Negotiation successful
0620 02:21:57.783565 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13133,"mutex_wait_us":422,"thread_start_us":12982,"threads_started":1}
I20260620 02:21:57.784354 17136 ts_manager.cc:194] Registered new tserver with Master: d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215)
I20260620 02:21:57.785251 17469 heartbeater.cc:347] Connected to a master server at 127.9.231.125:43761
I20260620 02:21:57.785463 17469 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:57.785809 17469 heartbeater.cc:511] Master 127.9.231.125:43761 requested a full tablet report, sending...
I20260620 02:21:57.786957 16947 ts_manager.cc:194] Registered new tserver with Master: d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215)
I20260620 02:21:57.866572 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16674
I20260620 02:21:57.884114 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:40697
--local_ip_for_outbound_sockets=127.9.231.67
--tserver_master_addrs=127.9.231.126:46181
--webserver_port=46033
--webserver_interface=127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab
--principal=oryx/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:21:58.102205 17317 heartbeater.cc:503] Master 127.9.231.126:46181 was elected leader, sending a full tablet report...
W20260620 02:21:58.270746 17480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:21:58.271224 17480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:21:58.271409 17480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:21:58.279386 17480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:21:58.279531 17480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:21:58.279719 17480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:21:58.291317 17480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab
--principal=oryx/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:40697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=46033
--tserver_master_addrs=127.9.231.126:46181,127.9.231.125:43761,127.9.231.124:39611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:21:58.292778 17480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:21:58.294238 17480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:21:58.309242 17485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:58.309264 17488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:21:58.309211 17486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:21:58.310127 17480 server_base.cc:1034] running on GCE node
I20260620 02:21:58.310853 17480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:21:58.312074 17480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:21:58.313450 17480 hybrid_clock.cc:648] HybridClock initialized: now 1781922118313369 us; error 69 us; skew 500 ppm
I20260620 02:21:58.314026 17480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:21:58 dist-test-slave-9vg1 krb5kdc[16298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922118, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:21:58.317970 17480 init.cc:377] Logged in from keytab as oryx/127.9.231.67@KRBTEST.COM (short username oryx)
I20260620 02:21:58.320070 17480 webserver.cc:460] Webserver started at http://127.9.231.67:46033/ using document root <none> and password file <none>
I20260620 02:21:58.320916 17480 fs_manager.cc:359] Metadata directory not provided
I20260620 02:21:58.321105 17480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:21:58.327750 17480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:21:58.331143 17495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:58.332803 17480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:21:58.333120 17480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "3bd6534c3827497ca34cb7d4cb3cb67c"
format_stamp: "Formatted at 2026-06-20 02:21:54 on dist-test-slave-9vg1"
server_key: "cb5d77a880463c61eb6aa7a8e0c5b6f4"
server_key_iv: "db3c6b72d0e71f9cdb497ea617ace896"
server_key_version: "encryptionkey@0"
I20260620 02:21:58.334259 17480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:21:58.361407 17480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:21:58.366387 17480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:21:58.366724 17480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:21:58.367976 17480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:21:58.369714 17480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:21:58.369822 17480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:58.369904 17480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:21:58.369963 17480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:21:58.407721 17480 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:40697
I20260620 02:21:58.407831 17608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:40697 every 8 connection(s)
I20260620 02:21:58.409469 17480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:21:58.415584 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 17480
Jun 20 02:21:58 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922118, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.124@KRBTEST.COM
Jun 20 02:21:58 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922118, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:21:58.448597 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.414132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.9.231.67:44439)
0620 02:21:58.415000 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.415051 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.418646 (+ 3595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.420337 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.420358 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.421562 (+ 1204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:58.422844 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.422888 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.425534 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.425544 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.428621 (+ 3077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.428643 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.430323 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.430331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.430790 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.431779 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:58.431817 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:58.435568 (+ 3751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:58.440980 (+ 5412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:58.440996 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:58.441027 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:58.441615 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:58.442187 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:58.442194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:58.442199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:58.442548 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:58.443496 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:58.443507 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:58.445117 (+ 1610us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.445510 (+ 393us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.446267 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":5,"thread_start_us":397,"threads_started":1}
I20260620 02:21:58.450047 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.413289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:44439 (local address 127.9.231.124:39611)
0620 02:21:58.413741 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.413747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.415258 (+ 1511us) server_negotiation.cc:407] Connection header received
0620 02:21:58.419241 (+ 3983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.419253 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.419426 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.419543 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.423301 (+ 3758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.425174 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.428882 (+ 3708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.429780 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.429901 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.435914 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.435923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.435929 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.435982 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.440743 (+ 4761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.441792 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.441804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.441807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.442046 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.442730 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.442737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.442740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.443233 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.443396 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.448267 (+ 4871us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.448615 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
Jun 20 02:21:58 dist-test-slave-9vg1 krb5kdc[16298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922118, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.125@KRBTEST.COM
I20260620 02:21:58.452111 17610 heartbeater.cc:347] Connected to a master server at 127.9.231.124:39611
I20260620 02:21:58.452512 17610 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:58.453653 17610 heartbeater.cc:511] Master 127.9.231.124:39611 requested a full tablet report, sending...
I20260620 02:21:58.456089 17136 ts_manager.cc:194] Registered new tserver with Master: 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67:40697)
I20260620 02:21:58.468537 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.416142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:44057 (local address 127.9.231.126:46181)
0620 02:21:58.429813 (+ 13671us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.429822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.429847 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:21:58.429914 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.429921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.430016 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.430137 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.432270 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.433612 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.436988 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.437786 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.437968 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.443434 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.443442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.443448 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.443500 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.462980 (+ 19480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.465127 (+ 2147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.465137 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.465144 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.465381 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.466034 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.466043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.466050 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.466550 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.467371 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.467604 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.467970 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13528,"thread_start_us":15155,"threads_started":1}
I20260620 02:21:58.468513 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.416002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:44057)
0620 02:21:58.427673 (+ 11671us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.427697 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.427839 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.430379 (+ 2540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.430388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.431241 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:58.432036 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.432059 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.433822 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.433835 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.436640 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.436661 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.438305 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.438314 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.438567 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.439337 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:58.439366 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:58.443150 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:58.463278 (+ 20128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:58.463288 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:58.463296 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:58.464090 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:58.465585 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:58.465593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:58.465601 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:58.465829 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:58.466729 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:58.466736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:58.466961 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.467169 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.467711 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11489,"mutex_wait_us":24,"thread_start_us":11269,"threads_started":1}
I20260620 02:21:58.470582 17609 heartbeater.cc:347] Connected to a master server at 127.9.231.126:46181
I20260620 02:21:58.470844 17609 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:58.471843 17609 heartbeater.cc:511] Master 127.9.231.126:46181 requested a full tablet report, sending...
I20260620 02:21:58.472474 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.431925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.9.231.67:37913)
0620 02:21:58.439149 (+ 7224us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.439177 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.439531 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.439990 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.439998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.440824 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:21:58.441319 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.441334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.443526 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.443536 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.445409 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.445428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.447215 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.447222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.447615 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.448755 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:21:58.448794 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:21:58.455765 (+ 6971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:21:58.463820 (+ 8055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:58.463830 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:58.463835 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:58.464901 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:58.465604 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:21:58.465614 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:21:58.465622 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:21:58.465915 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:21:58.467105 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:21:58.467115 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:21:58.469208 (+ 2093us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.469458 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.469954 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7010,"mutex_wait_us":56,"thread_start_us":6814,"threads_started":1}
I20260620 02:21:58.472975 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.428235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:37913 (local address 127.9.231.125:43761)
0620 02:21:58.431901 (+ 3666us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.431908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.439302 (+ 7394us) server_negotiation.cc:407] Connection header received
0620 02:21:58.439708 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.439714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.439807 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.439940 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.441541 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.443245 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.445587 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.446474 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.446712 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.457757 (+ 11045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.457765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.457770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.457825 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.463611 (+ 5786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.465084 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.465095 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.465102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.465374 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.466124 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.466134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.466140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.466714 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.466947 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.470790 (+ 3843us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.471236 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3532,"thread_start_us":3387,"threads_started":1}
I20260620 02:21:58.474033 16343 ts_manager.cc:194] Re-registered known tserver with Master: 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67:40697)
I20260620 02:21:58.474335 17611 heartbeater.cc:347] Connected to a master server at 127.9.231.125:43761
I20260620 02:21:58.474534 17611 heartbeater.cc:464] Registering TS with master...
I20260620 02:21:58.475206 17611 heartbeater.cc:511] Master 127.9.231.125:43761 requested a full tablet report, sending...
I20260620 02:21:58.475685 16343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:44057
I20260620 02:21:58.476663 16947 ts_manager.cc:194] Registered new tserver with Master: 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67:40697)
I20260620 02:21:58.485486 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.445877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.9.231.126:46181)
0620 02:21:58.450136 (+ 4259us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.450144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.450903 (+ 759us) server_negotiation.cc:407] Connection header received
0620 02:21:58.451295 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.451301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.451400 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.451493 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.454508 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.455848 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.458033 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.458861 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.458960 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.468342 (+ 9382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.468350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.468355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.468395 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.481079 (+ 12684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.482717 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.482722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.482725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.483008 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.483541 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.483547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.483551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.483929 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.484231 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.484710 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.484987 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4062,"thread_start_us":4538,"threads_started":1}
I20260620 02:21:58.491992 16343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:21:58.500954 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.451037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.9.231.125:43761)
0620 02:21:58.481286 (+ 30249us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.481296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.481335 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:21:58.481430 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.481438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.481593 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.481768 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.485449 (+ 3681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.486673 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.488837 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.490149 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.490312 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.493726 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.493732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.493735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.493777 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.497370 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.498243 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.498250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.498254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.498421 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.498851 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.498857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.498861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.499289 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.499570 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.500448 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.500673 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30051,"thread_start_us":30985,"threads_started":1}
W20260620 02:21:58.501852 16343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:21:58.512713 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.475913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.9.231.124:39611)
0620 02:21:58.476115 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.476123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.482921 (+ 6798us) server_negotiation.cc:407] Connection header received
0620 02:21:58.483165 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.483174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.483309 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.483411 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.486179 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.487219 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.489381 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.490220 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.490367 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.503293 (+ 12926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.503299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.503303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.503345 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.507180 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.508346 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.508352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.508356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.508595 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.509243 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.509249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.509254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.509772 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.510051 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.511617 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.512000 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:21:58.557955 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.544843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:40215 (local address 127.0.0.1:40058)
0620 02:21:58.545558 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.545581 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.545777 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.547006 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.547011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.547038 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:58.547518 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.547530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.548909 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.548914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.550798 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.550812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.556893 (+ 6081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.556900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.557132 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.557187 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.557291 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.557501 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":373,"threads_started":1}
I20260620 02:21:58.558923 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.545130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.9.231.66:40215)
0620 02:21:58.546370 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.546380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.546411 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:21:58.546517 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.546524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.546807 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.547025 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:58.547712 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.548711 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.550972 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.556653 (+ 5681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.556805 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.557027 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.558359 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.558586 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":1085,"threads_started":1}
I20260620 02:21:58.563108 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.546620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.9.231.67:40697)
0620 02:21:58.552651 (+ 6031us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.552664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.552711 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:21:58.552881 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.552894 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.553256 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.554505 (+ 1249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:58.555002 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.557254 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.560431 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.561687 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.561833 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.562060 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.562561 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.562747 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5781,"thread_start_us":5581,"threads_started":1}
I20260620 02:21:58.563390 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.546131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:40697 (local address 127.0.0.1:52662)
0620 02:21:58.551656 (+ 5525us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.551676 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.551812 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.553556 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.553565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.553604 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:58.554222 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.554239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.557454 (+ 3215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.557459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.560239 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.560255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.561994 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.562001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.562203 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.562335 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.562459 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.562620 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5362,"mutex_wait_us":28,"thread_start_us":5207,"threads_started":1}
I20260620 02:21:58.563618 17401 tablet_service.cc:1467] Processing CreateTablet for tablet 04c5af9864b84d1fb5047fac1c7034c0 (DEFAULT_TABLE table=test-table [id=f3a2c73e6f7f441c948c8a5cb2fcf87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:58.565322 17401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c5af9864b84d1fb5047fac1c7034c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:58.567564 17641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.551974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:41181 (local address 127.0.0.1:60958)
0620 02:21:58.556275 (+ 4301us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.556314 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.556502 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.557331 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.557339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.557373 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:58.557868 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.557884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.561300 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.561305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.563632 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.563645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.566260 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.566271 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.566506 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.566549 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.566672 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.566835 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4163,"mutex_wait_us":2,"thread_start_us":3983,"threads_started":1}
I20260620 02:21:58.567855 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.552185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.9.231.65:41181)
0620 02:21:58.555736 (+ 3551us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.555752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.556447 (+ 695us) server_negotiation.cc:407] Connection header received
0620 02:21:58.556705 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.556712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.557034 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.557288 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:58.559599 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.561125 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.563792 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.565937 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.566079 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.566283 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.566751 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.566912 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3288,"thread_start_us":3114,"threads_started":1}
I20260620 02:21:58.567615 17543 tablet_service.cc:1467] Processing CreateTablet for tablet 04c5af9864b84d1fb5047fac1c7034c0 (DEFAULT_TABLE table=test-table [id=f3a2c73e6f7f441c948c8a5cb2fcf87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:58.569504 17543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c5af9864b84d1fb5047fac1c7034c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:58.573446 17252 tablet_service.cc:1467] Processing CreateTablet for tablet 04c5af9864b84d1fb5047fac1c7034c0 (DEFAULT_TABLE table=test-table [id=f3a2c73e6f7f441c948c8a5cb2fcf87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:21:58.575214 17252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c5af9864b84d1fb5047fac1c7034c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:21:58.581100 17643 tablet_bootstrap.cc:492] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Bootstrap starting.
I20260620 02:21:58.584466 17643 tablet_bootstrap.cc:654] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:58.585860 17643 log.cc:824] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:58.589854 17643 tablet_bootstrap.cc:492] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: No bootstrap required, opened a new log
I20260620 02:21:58.590322 17643 ts_tablet_manager.cc:1397] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:21:58.590515 17645 tablet_bootstrap.cc:492] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Bootstrap starting.
I20260620 02:21:58.593437 17646 tablet_bootstrap.cc:492] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Bootstrap starting.
I20260620 02:21:58.595521 17645 tablet_bootstrap.cc:654] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:58.596992 17646 tablet_bootstrap.cc:654] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Neither blocks nor log segments found. Creating new log.
I20260620 02:21:58.596992 17645 log.cc:824] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:58.598232 17646 log.cc:824] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:21:58.601006 17645 tablet_bootstrap.cc:492] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: No bootstrap required, opened a new log
I20260620 02:21:58.601306 17645 ts_tablet_manager.cc:1397] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 02:21:58.601987 17646 tablet_bootstrap.cc:492] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: No bootstrap required, opened a new log
I20260620 02:21:58.602254 17646 ts_tablet_manager.cc:1397] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 02:21:58.602118 17643 raft_consensus.cc:348] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.602602 17643 raft_consensus.cc:374] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:58.602712 17643 raft_consensus.cc:729] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9732a2886db4b35ac7137c80866be27, State: Initialized, Role: FOLLOWER
I20260620 02:21:58.603467 17643 consensus_queue.cc:260] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.604617 17467 heartbeater.cc:503] Master 127.9.231.126:46181 was elected leader, sending a full tablet report...
I20260620 02:21:58.606756 17643 ts_tablet_manager.cc:1428] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260620 02:21:58.613773 17646 raft_consensus.cc:348] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.614315 17646 raft_consensus.cc:374] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:58.614423 17646 raft_consensus.cc:729] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf949289be24f38b43ca2d8ebe681d8, State: Initialized, Role: FOLLOWER
I20260620 02:21:58.615263 17646 consensus_queue.cc:260] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.616238 17645 raft_consensus.cc:348] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.616745 17645 raft_consensus.cc:374] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:21:58.616844 17645 raft_consensus.cc:729] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd6534c3827497ca34cb7d4cb3cb67c, State: Initialized, Role: FOLLOWER
I20260620 02:21:58.617424 17646 ts_tablet_manager.cc:1428] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260620 02:21:58.617517 17645 consensus_queue.cc:260] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.619223 17609 heartbeater.cc:503] Master 127.9.231.126:46181 was elected leader, sending a full tablet report...
I20260620 02:21:58.619993 17645 ts_tablet_manager.cc:1428] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260620 02:21:58.654649 17651 raft_consensus.cc:482] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:21:58.655025 17651 raft_consensus.cc:504] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.657366 17651 leader_election.cc:288] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215), bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
W20260620 02:21:58.663825 17612 tablet.cc:2369] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:58.664924 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.657945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:40215 (local address 127.9.231.67:49463)
0620 02:21:58.658204 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.658217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.658356 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.658758 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.658761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.658808 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:58.659234 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.659248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.660410 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.660417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.662286 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.662302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.663961 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.663967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.664200 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.664243 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.664338 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.664489 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":26}
I20260620 02:21:58.665397 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.658470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:49853 (local address 127.9.231.65:41181)
0620 02:21:58.658742 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.658749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.658790 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:21:58.658863 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.658869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.658933 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.659002 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:58.659583 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.660422 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.662497 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.663750 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.663928 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.664029 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.665011 (+ 982us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.665169 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 02:21:58.665720 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.657893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:49463 (local address 127.9.231.66:40215)
0620 02:21:58.658110 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.658118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.658296 (+ 178us) server_negotiation.cc:407] Connection header received
0620 02:21:58.658511 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.658540 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.658620 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.658716 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:21:58.659396 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.660273 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.662452 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.663564 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.663788 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.663867 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.665232 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.665393 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:21:58.665979 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.657697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:41181 (local address 127.9.231.67:49853)
0620 02:21:58.658014 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:58.658043 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:21:58.658272 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:21:58.659050 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:21:58.659116 (+ 66us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:21:58.659154 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:21:58.659379 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.659392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.660591 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.660598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.662311 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:21:58.662325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.664181 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.664187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:21:58.664488 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.664524 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:21:58.664625 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 02:21:58.664817 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":26}
I20260620 02:21:58.666795 17421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c5af9864b84d1fb5047fac1c7034c0" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9732a2886db4b35ac7137c80866be27" is_pre_election: true
I20260620 02:21:58.667338 17272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c5af9864b84d1fb5047fac1c7034c0" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf949289be24f38b43ca2d8ebe681d8" is_pre_election: true
I20260620 02:21:58.667543 17421 raft_consensus.cc:2454] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 0.
I20260620 02:21:58.667891 17272 raft_consensus.cc:2454] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 0.
I20260620 02:21:58.668428 17497 leader_election.cc:302] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd6534c3827497ca34cb7d4cb3cb67c, d9732a2886db4b35ac7137c80866be27; no voters:
I20260620 02:21:58.669054 17651 raft_consensus.cc:2781] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:21:58.669246 17651 raft_consensus.cc:482] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:21:58.669354 17651 raft_consensus.cc:3037] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:58.672945 17651 raft_consensus.cc:504] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.673722 17651 leader_election.cc:288] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 election: Requested vote from peers d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215), bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
I20260620 02:21:58.674329 17272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c5af9864b84d1fb5047fac1c7034c0" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf949289be24f38b43ca2d8ebe681d8"
I20260620 02:21:58.674566 17272 raft_consensus.cc:3037] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:58.675346 17421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c5af9864b84d1fb5047fac1c7034c0" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9732a2886db4b35ac7137c80866be27"
I20260620 02:21:58.675611 17421 raft_consensus.cc:3037] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:21:58.678035 17421 raft_consensus.cc:2454] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 1.
I20260620 02:21:58.678311 17272 raft_consensus.cc:2454] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 1.
I20260620 02:21:58.678630 17497 leader_election.cc:302] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd6534c3827497ca34cb7d4cb3cb67c, d9732a2886db4b35ac7137c80866be27; no voters:
I20260620 02:21:58.678987 17651 raft_consensus.cc:2781] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:21:58.679598 17651 raft_consensus.cc:686] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 LEADER]: Becoming Leader. State: Replica: 3bd6534c3827497ca34cb7d4cb3cb67c, State: Running, Role: LEADER
I20260620 02:21:58.680353 17651 consensus_queue.cc:237] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:21:58.686707 16343 catalog_manager.cc:5495] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67). New cstate: current_term: 1 leader_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20260620 02:21:58.719166 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.712999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.9.231.67:40697)
0620 02:21:58.713187 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.713192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.713246 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:21:58.713468 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.713472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.713556 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.713694 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:21:58.714177 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.714944 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.716747 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.717210 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.717330 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.717868 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:21:58.718441 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:21:58.718609 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.718741 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.718821 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:21:58.728972 17421 raft_consensus.cc:1261] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Refusing update from remote peer 3bd6534c3827497ca34cb7d4cb3cb67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:58.728951 17272 raft_consensus.cc:1261] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Refusing update from remote peer 3bd6534c3827497ca34cb7d4cb3cb67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:21:58.730222 17652 consensus_queue.cc:1035] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:21:58.730726 17651 consensus_queue.cc:1035] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:21:58.745688 17476 tablet.cc:2369] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:21:58.747264 17655 mvcc.cc:205] Tried to move back new op lower bound from 7298752998301130752 to 7298752998118907904. Current Snapshot: MvccSnapshot[applied={T|T < 7298752998301130752}]
I20260620 02:21:58.748023 17656 mvcc.cc:205] Tried to move back new op lower bound from 7298752998301130752 to 7298752998118907904. Current Snapshot: MvccSnapshot[applied={T|T < 7298752998301130752}]
I20260620 02:21:58.777325 16343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:58.777621 16343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:21:58.785115 16343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d6be40cafc84abbbaaeeecf9f4c55a2: Sending DeleteTablet for 3 replicas of tablet 04c5af9864b84d1fb5047fac1c7034c0
I20260620 02:21:58.786645 17401 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c5af9864b84d1fb5047fac1c7034c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:58 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:40058
I20260620 02:21:58.787312 17252 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c5af9864b84d1fb5047fac1c7034c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:58 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:60958
I20260620 02:21:58.787704 17543 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c5af9864b84d1fb5047fac1c7034c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:21:58 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:52662
I20260620 02:21:58.797112 17667 tablet_replica.cc:330] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: stopping tablet replica
I20260620 02:21:58.799448 17667 raft_consensus.cc:2229] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:21:58.800330 17667 raft_consensus.cc:2258] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:58.801237 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16311
I20260620 02:21:58.802897 16373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:21:58.803978 17665 tablet_replica.cc:330] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: stopping tablet replica
I20260620 02:21:58.807797 17665 raft_consensus.cc:2229] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:58.808712 17665 raft_consensus.cc:2258] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:58.814687 17667 ts_tablet_manager.cc:1905] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:58.814945 17665 ts_tablet_manager.cc:1905] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:58.819296 17666 tablet_replica.cc:330] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: stopping tablet replica
I20260620 02:21:58.820379 17666 raft_consensus.cc:2229] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:21:58.823007 17666 raft_consensus.cc:2258] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:21:58.825659 17665 ts_tablet_manager.cc:1918] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:58.825870 17665 log.cc:1196] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/04c5af9864b84d1fb5047fac1c7034c0
I20260620 02:21:58.826452 17665 ts_tablet_manager.cc:1939] T 04c5af9864b84d1fb5047fac1c7034c0 P d9732a2886db4b35ac7137c80866be27: Deleting consensus metadata
I20260620 02:21:58.828284 17666 ts_tablet_manager.cc:1905] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:21:58.832978 17667 ts_tablet_manager.cc:1918] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:58.833204 17667 log.cc:1196] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/04c5af9864b84d1fb5047fac1c7034c0
I20260620 02:21:58.833841 17667 ts_tablet_manager.cc:1939] T 04c5af9864b84d1fb5047fac1c7034c0 P 3bd6534c3827497ca34cb7d4cb3cb67c: Deleting consensus metadata
I20260620 02:21:58.837874 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.790079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.9.231.125:43761)
0620 02:21:58.790280 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.790286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.802915 (+ 12629us) server_negotiation.cc:407] Connection header received
0620 02:21:58.803933 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.803939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.804034 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.804155 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.810435 (+ 6280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.811772 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.819898 (+ 8126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.820594 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.820750 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.829370 (+ 8620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.829378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.829382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.829435 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.833200 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.834517 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.834523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.834527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.834722 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.836256 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.836263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.836266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.836620 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.836906 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.837260 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.837512 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:21:58.847790 17666 ts_tablet_manager.cc:1918] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:21:58.848119 17666 log.cc:1196] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/04c5af9864b84d1fb5047fac1c7034c0
I20260620 02:21:58.848896 17666 ts_tablet_manager.cc:1939] T 04c5af9864b84d1fb5047fac1c7034c0 P bbf949289be24f38b43ca2d8ebe681d8: Deleting consensus metadata
I20260620 02:21:58.857654 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.800286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.9.231.124:39611)
0620 02:21:58.800486 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.800493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.837947 (+ 37454us) server_negotiation.cc:407] Connection header received
0620 02:21:58.838230 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.838236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.838322 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.838402 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.840748 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.841925 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.844526 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.845265 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.845386 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:58.849647 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:58.849656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:58.849661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:58.849725 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:58.854078 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.855182 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.855189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.855193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.855364 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:58.855884 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:58.855889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:58.855893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:58.856292 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:58.856599 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:58.856988 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 02:21:58.857234 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:21:58.959453 16328 catalog_manager.cc:4841] TS d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215): tablet 04c5af9864b84d1fb5047fac1c7034c0 (table test-table [id=f3a2c73e6f7f441c948c8a5cb2fcf87f]) successfully deleted
I20260620 02:21:58.959648 16329 catalog_manager.cc:4841] TS bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181): tablet 04c5af9864b84d1fb5047fac1c7034c0 (table test-table [id=f3a2c73e6f7f441c948c8a5cb2fcf87f]) successfully deleted
I20260620 02:21:58.959976 16329 catalog_manager.cc:4841] TS 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67:40697): tablet 04c5af9864b84d1fb5047fac1c7034c0 (table test-table [id=f3a2c73e6f7f441c948c8a5cb2fcf87f]) successfully deleted
I20260620 02:21:58.960299 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16311
I20260620 02:21:59.007617 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.989605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.9.231.125:43761)
0620 02:21:58.989875 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.989885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.989915 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:21:58.990087 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.990096 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.990193 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.990319 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.992525 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.993756 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.996008 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.996740 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.996879 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:59.000607 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:59.000614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:59.000619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:59.000671 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:59.004331 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:59.005230 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:59.005236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:59.005240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:59.005410 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:59.005811 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:59.005816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:59.005819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:59.006139 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:59.006405 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:59.006763 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 02:21:59.007009 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:21:59.010272 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:58.990739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.9.231.124:39611)
0620 02:21:58.991205 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 02:21:58.991209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:21:58.991469 (+ 260us) server_negotiation.cc:407] Connection header received
0620 02:21:58.991667 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:21:58.991672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:21:58.991745 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:21:58.991856 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:21:58.994078 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.994936 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.996872 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:21:58.997605 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:21:58.997747 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:21:59.002856 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:21:59.002864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:21:59.002869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:21:59.002919 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:21:59.006409 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:59.007357 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:59.007362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:59.007365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:59.007497 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:21:59.007942 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:21:59.007948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:21:59.007951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:21:59.008335 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:21:59.008653 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 02:21:59.009660 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0620 02:21:59.009907 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260620 02:21:59.767872 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:59.766746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.65:47021)
0620 02:21:59.767524 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:59.767669 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":549,"threads_started":1}
W20260620 02:21:59.768738 17317 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:46181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
I20260620 02:21:59.771704 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:59.770799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:34199)
0620 02:21:59.771308 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:59.771448 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
W20260620 02:21:59.772102 17609 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:46181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
I20260620 02:21:59.777292 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:21:59.776243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.66:56477)
0620 02:21:59.776812 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 02:21:59.777061 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
W20260620 02:21:59.777750 17467 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:46181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
I20260620 02:22:00.287249 17679 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d6be40cafc84abbbaaeeecf9f4c55a2)
I20260620 02:22:00.287304 17680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d6be40cafc84abbbaaeeecf9f4c55a2)
I20260620 02:22:00.287590 17680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } }
I20260620 02:22:00.287608 17679 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } }
I20260620 02:22:00.289191 17680 leader_election.cc:288] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d6be40cafc84abbbaaeeecf9f4c55a2 (127.9.231.126:46181), f6513d315fac4b87baaeb279a47978d5 (127.9.231.124:39611)
I20260620 02:22:00.289207 17679 leader_election.cc:288] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d6be40cafc84abbbaaeeecf9f4c55a2 (127.9.231.126:46181), 42f9acbf4c54411481ad5350040af0f8 (127.9.231.125:43761)
I20260620 02:22:00.292135 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.289343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42036)
0620 02:22:00.291600 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.291827 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2104,"thread_start_us":1943,"threads_started":1}
I20260620 02:22:00.293002 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.289344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42034)
0620 02:22:00.292478 (+ 3134us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.292680 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2987,"thread_start_us":2830,"threads_started":1}
W20260620 02:22:00.293466 16931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
W20260620 02:22:00.293563 17120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
I20260620 02:22:00.304248 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.291838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:39611 (local address 127.0.0.1:49952)
0620 02:22:00.292429 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.292460 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:00.292636 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:00.294021 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:00.294029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:00.294066 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:00.294429 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.294444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.297266 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.297276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.300523 (+ 3247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.300562 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.302880 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.302888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.303220 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.303287 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 02:22:00.303413 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 02:22:00.303665 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":2,"thread_start_us":1185,"threads_started":1}
I20260620 02:22:00.304791 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.293237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43761 (local address 127.0.0.1:57712)
0620 02:22:00.293709 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.293729 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:00.293855 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:00.294601 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:00.294609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:00.294647 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:00.295023 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.295040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.297557 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.297566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.300727 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.300759 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.303680 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.303691 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.303989 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.304049 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 02:22:00.304166 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:22:00.304387 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364}
I20260620 02:22:00.305513 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.302986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42044)
0620 02:22:00.305107 (+ 2121us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.305253 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1983,"threads_started":1}
I20260620 02:22:00.305598 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.292160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.9.231.124:39611)
0620 02:22:00.293596 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.293602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.293621 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:22:00.293701 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.293710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.293815 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.294641 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:00.295035 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.297005 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.300944 (+ 3939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.302642 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.304591 (+ 1949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.304795 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.304867 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.305072 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291,"thread_start_us":1331,"threads_started":1}
I20260620 02:22:00.306378 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.292699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.9.231.125:43761)
0620 02:22:00.293471 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.293477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.293970 (+ 493us) server_negotiation.cc:407] Connection header received
0620 02:22:00.294044 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.294052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.294176 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.294289 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:00.295399 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.297258 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.301064 (+ 3806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.303157 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.303334 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.303530 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.305706 (+ 2176us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.305916 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"spinlock_wait_cycles":3968,"thread_start_us":1043,"threads_started":1}
I20260620 02:22:00.306828 17146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42f9acbf4c54411481ad5350040af0f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6513d315fac4b87baaeb279a47978d5" is_pre_election: true
I20260620 02:22:00.307222 17146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42f9acbf4c54411481ad5350040af0f8 in term 1.
I20260620 02:22:00.307403 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.306968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42050)
0620 02:22:00.307164 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.307250 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:22:00.307533 17120 leader_election.cc:334] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d6be40cafc84abbbaaeeecf9f4c55a2 (127.9.231.126:46181): Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
W20260620 02:22:00.307644 16931 leader_election.cc:334] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d6be40cafc84abbbaaeeecf9f4c55a2 (127.9.231.126:46181): Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
I20260620 02:22:00.308038 16931 leader_election.cc:302] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42f9acbf4c54411481ad5350040af0f8, f6513d315fac4b87baaeb279a47978d5; no voters: 9d6be40cafc84abbbaaeeecf9f4c55a2
I20260620 02:22:00.308410 17680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:22:00.308368 16956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6513d315fac4b87baaeb279a47978d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42f9acbf4c54411481ad5350040af0f8" is_pre_election: true
I20260620 02:22:00.308561 17680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9d6be40cafc84abbbaaeeecf9f4c55a2)
I20260620 02:22:00.308688 17680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:22:00.310951 17680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } }
I20260620 02:22:00.311770 16956 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6513d315fac4b87baaeb279a47978d5 in current term 2: Already voted for candidate 42f9acbf4c54411481ad5350040af0f8 in this term.
I20260620 02:22:00.312047 17680 leader_election.cc:288] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [CANDIDATE]: Term 2 election: Requested vote from peers 9d6be40cafc84abbbaaeeecf9f4c55a2 (127.9.231.126:46181), f6513d315fac4b87baaeb279a47978d5 (127.9.231.124:39611)
I20260620 02:22:00.312682 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.312199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42060)
0620 02:22:00.312388 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.312503 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 02:22:00.313246 17120 leader_election.cc:302] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6513d315fac4b87baaeb279a47978d5; no voters: 42f9acbf4c54411481ad5350040af0f8, 9d6be40cafc84abbbaaeeecf9f4c55a2
I20260620 02:22:00.313740 17146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42f9acbf4c54411481ad5350040af0f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6513d315fac4b87baaeb279a47978d5"
I20260620 02:22:00.313918 17679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:22:00.314744 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.314338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42074)
0620 02:22:00.314527 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.314598 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 02:22:00.315099 16931 leader_election.cc:334] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9d6be40cafc84abbbaaeeecf9f4c55a2 (127.9.231.126:46181): Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
I20260620 02:22:00.316794 17679 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:22:00.319350 17146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42f9acbf4c54411481ad5350040af0f8 in term 2.
I20260620 02:22:00.320411 16931 leader_election.cc:302] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42f9acbf4c54411481ad5350040af0f8, f6513d315fac4b87baaeb279a47978d5; no voters: 9d6be40cafc84abbbaaeeecf9f4c55a2
I20260620 02:22:00.320812 17680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:22:00.321400 17680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [term 2 LEADER]: Becoming Leader. State: Replica: 42f9acbf4c54411481ad5350040af0f8, State: Running, Role: LEADER
I20260620 02:22:00.321848 17680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } }
I20260620 02:22:00.323926 17689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f9acbf4c54411481ad5350040af0f8. Latest consensus state: current_term: 2 leader_uuid: "42f9acbf4c54411481ad5350040af0f8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:22:00.324280 17689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:00.324931 17691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:22:00.329190 17691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:22:00.330176 17691 catalog_manager.cc:1259] Loaded cluster ID: b84011e6ba2e471e92c06586c49e4a4f
I20260620 02:22:00.330307 17691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:22:00.331604 17691 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:22:00.332473 17691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Loaded TSK: 0
I20260620 02:22:00.333384 17691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:22:00.385810 16946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:22:00.393577 16946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:22:00.398492 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.398047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42096)
0620 02:22:00.398216 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.398335 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 02:22:00.399081 17146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [term 2 FOLLOWER]: Refusing update from remote peer 42f9acbf4c54411481ad5350040af0f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 02:22:00.399847 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.399357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42102)
0620 02:22:00.399555 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.399669 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 02:22:00.399978 17689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 02:22:00.400285 16931 consensus_peers.cc:487] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 -> Peer 9d6be40cafc84abbbaaeeecf9f4c55a2 (127.9.231.126:46181): Couldn't send request to peer 9d6be40cafc84abbbaaeeecf9f4c55a2. Status: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:22:00.403348 17679 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f9acbf4c54411481ad5350040af0f8. Latest consensus state: current_term: 2 leader_uuid: "42f9acbf4c54411481ad5350040af0f8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:22:00.403770 17679 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:22:00.405781 17689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "42f9acbf4c54411481ad5350040af0f8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:22:00.406288 17689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:00.406598 17679 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "42f9acbf4c54411481ad5350040af0f8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6be40cafc84abbbaaeeecf9f4c55a2" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 46181 } } peers { permanent_uuid: "42f9acbf4c54411481ad5350040af0f8" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43761 } attrs { promote: false } } peers { permanent_uuid: "f6513d315fac4b87baaeb279a47978d5" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 39611 } attrs { promote: false } } }
I20260620 02:22:00.406898 17679 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513d315fac4b87baaeb279a47978d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:22:00.425370 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.416692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:40215 (local address 127.0.0.1:40064)
0620 02:22:00.416976 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.416998 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:00.417125 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:00.418258 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:00.418263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:00.418287 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:00.418488 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.418501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.419987 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.419993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.422547 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.422563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.424361 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.424367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.424557 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.424606 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 02:22:00.424729 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:22:00.425015 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":130}
I20260620 02:22:00.426376 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.417397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.9.231.66:40215)
0620 02:22:00.417878 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.417887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.417924 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:22:00.418006 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.418012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.418088 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.418240 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:00.418679 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.419825 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.422768 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.424162 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.424323 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.424454 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.425796 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.425984 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":347,"threads_started":1}
I20260620 02:22:00.426458 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.416999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:40697 (local address 127.0.0.1:52694)
0620 02:22:00.417195 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.417217 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:00.417337 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:00.419794 (+ 2457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:00.419803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:00.419846 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:00.420489 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.420501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.421972 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.421977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.424248 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.424265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.425778 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.425785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.425909 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.425936 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:22:00.425991 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 02:22:00.426089 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":59,"spinlock_wait_cycles":9216}
I20260620 02:22:00.427470 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.417843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.9.231.67:40697)
0620 02:22:00.419447 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.419453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.419473 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:22:00.419551 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.419556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.419638 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.420152 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:00.420664 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.421824 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.424455 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.425518 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.426828 (+ 1310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.426940 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.426996 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.427163 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":1377,"threads_started":1}
I20260620 02:22:00.427356 17401 tablet_service.cc:1467] Processing CreateTablet for tablet ba6007df03ff4da3b4e2d291da3bab77 (DEFAULT_TABLE table=test-table [id=b1922138ff14409298e5e2b4ed68f0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:00.427645 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.417621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.9.231.65:41181)
0620 02:22:00.418162 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.418168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.419381 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 02:22:00.419465 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.419469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.419540 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.419671 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:00.420705 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.421940 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.424551 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.425985 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.426149 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.426261 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.427163 (+ 902us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.427315 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":834,"threads_started":1}
I20260620 02:22:00.427917 17401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6007df03ff4da3b4e2d291da3bab77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:00.428203 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.417407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:41181 (local address 127.0.0.1:60966)
0620 02:22:00.419087 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.419111 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:00.419267 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:00.419705 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:00.419710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:00.419736 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:00.420354 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.420372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.422131 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.422137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.424341 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:00.424354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.426475 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.426487 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:00.426710 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.426749 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:22:00.426848 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:22:00.427005 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1490,"mutex_wait_us":1,"spinlock_wait_cycles":2304,"thread_start_us":1297,"threads_started":1}
I20260620 02:22:00.428184 17543 tablet_service.cc:1467] Processing CreateTablet for tablet ba6007df03ff4da3b4e2d291da3bab77 (DEFAULT_TABLE table=test-table [id=b1922138ff14409298e5e2b4ed68f0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:00.428613 17543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6007df03ff4da3b4e2d291da3bab77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:00.432071 17252 tablet_service.cc:1467] Processing CreateTablet for tablet ba6007df03ff4da3b4e2d291da3bab77 (DEFAULT_TABLE table=test-table [id=b1922138ff14409298e5e2b4ed68f0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:00.432595 17252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6007df03ff4da3b4e2d291da3bab77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:00.434054 17702 tablet_bootstrap.cc:492] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: Bootstrap starting.
I20260620 02:22:00.435232 17703 tablet_bootstrap.cc:492] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: Bootstrap starting.
I20260620 02:22:00.436232 17702 tablet_bootstrap.cc:654] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:00.437389 17703 tablet_bootstrap.cc:654] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:00.439322 17706 tablet_bootstrap.cc:492] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Bootstrap starting.
I20260620 02:22:00.439811 17702 tablet_bootstrap.cc:492] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: No bootstrap required, opened a new log
I20260620 02:22:00.440032 17702 ts_tablet_manager.cc:1397] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 02:22:00.441288 17706 tablet_bootstrap.cc:654] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:00.441466 17702 raft_consensus.cc:348] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.441831 17702 raft_consensus.cc:374] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:00.441956 17702 raft_consensus.cc:729] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9732a2886db4b35ac7137c80866be27, State: Initialized, Role: FOLLOWER
I20260620 02:22:00.442497 17702 consensus_queue.cc:260] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.446043 17702 ts_tablet_manager.cc:1428] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 02:22:00.446365 17703 tablet_bootstrap.cc:492] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: No bootstrap required, opened a new log
I20260620 02:22:00.446555 17703 ts_tablet_manager.cc:1397] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260620 02:22:00.447431 17703 raft_consensus.cc:348] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.447638 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.445969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.66:39605)
0620 02:22:00.447161 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.447354 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":865,"threads_started":1}
I20260620 02:22:00.447710 17703 raft_consensus.cc:374] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:00.447808 17703 raft_consensus.cc:729] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd6534c3827497ca34cb7d4cb3cb67c, State: Initialized, Role: FOLLOWER
I20260620 02:22:00.448056 17703 consensus_queue.cc:260] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.450335 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.449187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:56497)
0620 02:22:00.449763 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.449996 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":612,"threads_started":1}
I20260620 02:22:00.451231 17469 heartbeater.cc:503] Master 127.9.231.125:43761 was elected leader, sending a full tablet report...
I20260620 02:22:00.451761 17703 ts_tablet_manager.cc:1428] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:22:00.452237 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.448402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.66:46629)
0620 02:22:00.451111 (+ 2709us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.451215 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2555}
I20260620 02:22:00.453099 17706 tablet_bootstrap.cc:492] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: No bootstrap required, opened a new log
I20260620 02:22:00.453275 17706 ts_tablet_manager.cc:1397] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260620 02:22:00.454141 17706 raft_consensus.cc:348] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.454398 17706 raft_consensus.cc:374] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:00.454509 17706 raft_consensus.cc:729] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf949289be24f38b43ca2d8ebe681d8, State: Initialized, Role: FOLLOWER
I20260620 02:22:00.454793 17611 heartbeater.cc:503] Master 127.9.231.125:43761 was elected leader, sending a full tablet report...
I20260620 02:22:00.454840 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.454317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:45903)
0620 02:22:00.454598 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.454698 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 02:22:00.454768 17706 consensus_queue.cc:260] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.455749 17706 ts_tablet_manager.cc:1428] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 02:22:00.457261 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.456224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.65:45445)
0620 02:22:00.456776 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.456900 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":751,"threads_started":1}
I20260620 02:22:00.462307 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.458012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.65:57783)
0620 02:22:00.461832 (+ 3820us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.462072 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3624,"thread_start_us":3778,"threads_started":1}
I20260620 02:22:00.560655 17710 raft_consensus.cc:482] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:22:00.560957 17710 raft_consensus.cc:504] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.562072 17710 leader_election.cc:288] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215), bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
I20260620 02:22:00.562700 17421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6007df03ff4da3b4e2d291da3bab77" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9732a2886db4b35ac7137c80866be27" is_pre_election: true
I20260620 02:22:00.563082 17421 raft_consensus.cc:2454] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 0.
I20260620 02:22:00.563658 17271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6007df03ff4da3b4e2d291da3bab77" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf949289be24f38b43ca2d8ebe681d8" is_pre_election: true
I20260620 02:22:00.563800 17497 leader_election.cc:302] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd6534c3827497ca34cb7d4cb3cb67c, d9732a2886db4b35ac7137c80866be27; no voters:
I20260620 02:22:00.564028 17271 raft_consensus.cc:2454] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 0.
I20260620 02:22:00.564366 17710 raft_consensus.cc:2781] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:22:00.564514 17710 raft_consensus.cc:482] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:22:00.564637 17710 raft_consensus.cc:3037] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:00.567893 17710 raft_consensus.cc:504] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.568760 17710 leader_election.cc:288] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 election: Requested vote from peers d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215), bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181)
I20260620 02:22:00.569525 17271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6007df03ff4da3b4e2d291da3bab77" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf949289be24f38b43ca2d8ebe681d8"
I20260620 02:22:00.569525 17421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6007df03ff4da3b4e2d291da3bab77" candidate_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9732a2886db4b35ac7137c80866be27"
I20260620 02:22:00.569795 17271 raft_consensus.cc:3037] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:00.569794 17421 raft_consensus.cc:3037] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:22:00.573233 17325 tablet.cc:2369] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:22:00.573341 17271 raft_consensus.cc:2454] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 1.
I20260620 02:22:00.573333 17421 raft_consensus.cc:2454] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd6534c3827497ca34cb7d4cb3cb67c in term 1.
I20260620 02:22:00.573916 17498 leader_election.cc:302] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd6534c3827497ca34cb7d4cb3cb67c, bbf949289be24f38b43ca2d8ebe681d8; no voters:
I20260620 02:22:00.574385 17710 raft_consensus.cc:2781] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:00.574826 17710 raft_consensus.cc:686] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 LEADER]: Becoming Leader. State: Replica: 3bd6534c3827497ca34cb7d4cb3cb67c, State: Running, Role: LEADER
I20260620 02:22:00.575196 17710 consensus_queue.cc:237] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } }
I20260620 02:22:00.576231 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.575664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:41579)
0620 02:22:00.575905 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.576043 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:22:00.578763 16946 catalog_manager.cc:5495] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67). New cstate: current_term: 1 leader_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bd6534c3827497ca34cb7d4cb3cb67c" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20260620 02:22:00.629716 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.624826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.9.231.67:40697)
0620 02:22:00.625158 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.625165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.625187 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:22:00.625274 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.625280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.625359 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.625465 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:00.625841 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.626585 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.628005 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.628419 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.628565 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.628973 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:00.629086 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:00.629187 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.629341 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.629428 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:22:00.633495 17421 raft_consensus.cc:1261] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Refusing update from remote peer 3bd6534c3827497ca34cb7d4cb3cb67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:00.633522 17271 raft_consensus.cc:1261] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Refusing update from remote peer 3bd6534c3827497ca34cb7d4cb3cb67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:00.634253 17710 consensus_queue.cc:1035] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf949289be24f38b43ca2d8ebe681d8" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:00.634573 17715 consensus_queue.cc:1035] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9732a2886db4b35ac7137c80866be27" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:00.636602 17319 heartbeater.cc:503] Master 127.9.231.125:43761 was elected leader, sending a full tablet report...
I20260620 02:22:00.637765 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.637191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.66:50831)
0620 02:22:00.637367 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.637522 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 02:22:00.637982 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.637305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.65:46199)
0620 02:22:00.637558 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.637702 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:22:00.643554 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.642752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:41645)
0620 02:22:00.643233 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.643339 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364}
I20260620 02:22:00.644100 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.643600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.66:35317)
0620 02:22:00.643838 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.643948 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 02:22:00.650194 16946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:00.650434 16946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:00.659426 16946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42f9acbf4c54411481ad5350040af0f8: Sending DeleteTablet for 3 replicas of tablet ba6007df03ff4da3b4e2d291da3bab77
I20260620 02:22:00.659535 17252 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6007df03ff4da3b4e2d291da3bab77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:00 UTC) from {username='oryx', principal='oryx/127.9.231.125@KRBTEST.COM'} at 127.0.0.1:60966
I20260620 02:22:00.660604 17251 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6007df03ff4da3b4e2d291da3bab77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:00 UTC) from {username='oryx', principal='oryx/127.9.231.125@KRBTEST.COM'} at 127.0.0.1:60966
I20260620 02:22:00.660693 17401 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6007df03ff4da3b4e2d291da3bab77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:00 UTC) from {username='oryx', principal='oryx/127.9.231.125@KRBTEST.COM'} at 127.0.0.1:40064
I20260620 02:22:00.661345 17543 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6007df03ff4da3b4e2d291da3bab77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:00 UTC) from {username='oryx', principal='oryx/127.9.231.125@KRBTEST.COM'} at 127.0.0.1:52694
I20260620 02:22:00.661909 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.660434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.67:35197)
0620 02:22:00.661359 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.661510 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":744}
I20260620 02:22:00.662652 17728 tablet_replica.cc:330] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: stopping tablet replica
I20260620 02:22:00.664336 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.663838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.9.231.65:42061)
0620 02:22:00.664051 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.664154 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 02:22:00.664556 17728 raft_consensus.cc:2229] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:22:00.665200 17728 raft_consensus.cc:2258] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:00.666785 17728 ts_tablet_manager.cc:1905] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:00.669080 17727 tablet_replica.cc:330] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: stopping tablet replica
I20260620 02:22:00.669320 17727 raft_consensus.cc:2229] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:00.669677 17727 raft_consensus.cc:2258] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:00.671665 17727 ts_tablet_manager.cc:1905] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:00.676057 17728 ts_tablet_manager.cc:1918] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:00.676254 17728 log.cc:1196] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/ba6007df03ff4da3b4e2d291da3bab77
I20260620 02:22:00.676712 17728 ts_tablet_manager.cc:1939] T ba6007df03ff4da3b4e2d291da3bab77 P 3bd6534c3827497ca34cb7d4cb3cb67c: Deleting consensus metadata
I20260620 02:22:00.677889 16933 catalog_manager.cc:4841] TS 3bd6534c3827497ca34cb7d4cb3cb67c (127.9.231.67:40697): tablet ba6007df03ff4da3b4e2d291da3bab77 (table test-table [id=b1922138ff14409298e5e2b4ed68f0e7]) successfully deleted
I20260620 02:22:00.679811 17727 ts_tablet_manager.cc:1918] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:00.680003 17727 log.cc:1196] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/ba6007df03ff4da3b4e2d291da3bab77
I20260620 02:22:00.680033 17726 tablet_replica.cc:330] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: stopping tablet replica
I20260620 02:22:00.680263 17726 raft_consensus.cc:2229] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:00.680444 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 17113
I20260620 02:22:00.680559 17727 ts_tablet_manager.cc:1939] T ba6007df03ff4da3b4e2d291da3bab77 P d9732a2886db4b35ac7137c80866be27: Deleting consensus metadata
I20260620 02:22:00.680617 17726 raft_consensus.cc:2258] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:00.681895 17726 ts_tablet_manager.cc:1905] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:00.683367 16932 catalog_manager.cc:4841] TS d9732a2886db4b35ac7137c80866be27 (127.9.231.66:40215): tablet ba6007df03ff4da3b4e2d291da3bab77 (table test-table [id=b1922138ff14409298e5e2b4ed68f0e7]) successfully deleted
I20260620 02:22:00.689239 17726 ts_tablet_manager.cc:1918] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:00.689467 17726 log.cc:1196] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/ba6007df03ff4da3b4e2d291da3bab77
I20260620 02:22:00.689904 17726 ts_tablet_manager.cc:1939] T ba6007df03ff4da3b4e2d291da3bab77 P bbf949289be24f38b43ca2d8ebe681d8: Deleting consensus metadata
I20260620 02:22:00.691269 16933 catalog_manager.cc:4841] TS bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181): tablet ba6007df03ff4da3b4e2d291da3bab77 (table test-table [id=b1922138ff14409298e5e2b4ed68f0e7]) successfully deleted
W20260620 02:22:00.692360 16933 catalog_manager.cc:4808] TS bbf949289be24f38b43ca2d8ebe681d8 (127.9.231.65:41181): delete failed for tablet ba6007df03ff4da3b4e2d291da3bab77 because the tablet was not found. No further retry: Not found: Tablet not found: ba6007df03ff4da3b4e2d291da3bab77
I20260620 02:22:00.701552 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.681702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.9.231.65:41181)
0620 02:22:00.681928 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.681934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.682595 (+ 661us) server_negotiation.cc:407] Connection header received
0620 02:22:00.682878 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.682884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.684270 (+ 1386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.684374 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:00.686617 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.687895 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.690120 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.690808 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.690980 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.694882 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:00.694903 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:00.694923 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:00.694965 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:00.698300 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:00.699414 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:00.699423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:00.699426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:00.699585 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:00.700057 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:00.700060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:00.700062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:00.700453 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:00.700700 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.701092 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.701307 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:22:00.702427 17312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:00.800792 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 17113
I20260620 02:22:00.820997 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 17329
I20260620 02:22:00.835667 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.821842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.9.231.66:40215)
0620 02:22:00.822039 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.822044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.822063 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:22:00.822182 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.822186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.822251 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.822343 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:00.823882 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.824593 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.825840 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.826292 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.826425 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.829436 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:00.829453 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:00.829470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:00.829527 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:00.832714 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:00.833637 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:00.833654 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:00.833657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:00.833807 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:00.834152 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:00.834155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:00.834157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:00.834599 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:00.834837 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.835195 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.835390 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:22:00.836519 17462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:00.889155 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.888685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42118)
0620 02:22:00.888878 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.888988 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:22:00.890309 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.889917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:46181 (local address 127.0.0.1:42124)
0620 02:22:00.890079 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:00.890187 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:22:00.944420 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 17329
I20260620 02:22:00.964417 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 17480
I20260620 02:22:00.979874 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:00.965487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.9.231.67:40697)
0620 02:22:00.965789 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:00.965798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:00.965843 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:22:00.966146 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:00.966150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:00.966231 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:00.966351 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:00.967905 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.968632 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.969956 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:00.970375 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:00.970573 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:00.973519 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:00.973537 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:00.973555 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:00.973598 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:00.976656 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:00.977498 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:00.977506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:00.977508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:00.977663 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:00.978059 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:00.978064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:00.978066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:00.978531 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:00.978799 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:00.979278 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 02:22:00.979573 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260620 02:22:00.980635 17604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:01.107369 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 17480
I20260620 02:22:01.131896 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16809
I20260620 02:22:01.133250 16977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:01.267359 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16809
W20260620 02:22:01.290244 17121 connection.cc:535] server connection from 127.0.0.1:49952 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:22:01.290628 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 16911
I20260620 02:22:01.292158 17166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:01.413337 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 16911
2026-06-20T02:22:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10438 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:22:01 dist-test-slave-9vg1 krb5kdc[17738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:22:01 dist-test-slave-9vg1 krb5kdc[17738](info): set up 2 sockets
Jun 20 02:22:01 dist-test-slave-9vg1 krb5kdc[17738](info): commencing operation
krb5kdc: starting...
W20260620 02:22:02.549331 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:22:02 dist-test-slave-9vg1 krb5kdc[17738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:22:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab.
Entry for principal oryx/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab.
I20260620 02:22:02.796417 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:39019
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab
--principal=oryx/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:03.175762 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:03.176113 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:03.176316 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:03.184510 17751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:22:03.185273 17751 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:22:03.185379 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:03.185493 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:03.185571 17751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:22:03.185712 17751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:22:03.199900 17751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:39019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.126.keytab
--principal=oryx/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:03.201617 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:03.203748 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:03.220013 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:03.220417 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:03.219991 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:03.221484 17751 server_base.cc:1034] running on GCE node
I20260620 02:22:03.222766 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:03.225003 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:03.226348 17751 hybrid_clock.cc:648] HybridClock initialized: now 1781922123226272 us; error 56 us; skew 500 ppm
I20260620 02:22:03.226774 17751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:03 dist-test-slave-9vg1 krb5kdc[17738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922123, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:03.231698 17751 init.cc:377] Logged in from keytab as oryx/127.9.231.126@KRBTEST.COM (short username oryx)
I20260620 02:22:03.233779 17751 webserver.cc:460] Webserver started at http://127.9.231.126:33517/ using document root <none> and password file <none>
I20260620 02:22:03.234601 17751 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:03.234700 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:03.235009 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:03.238880 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "b607451c74364485a977b236ec946af4"
format_stamp: "Formatted at 2026-06-20 02:22:03 on dist-test-slave-9vg1"
server_key: "7f216e00f164c416aa4392833bb316f2"
server_key_iv: "bb488feafbc43eab7b90789583f5f427"
server_key_version: "encryptionkey@0"
I20260620 02:22:03.240038 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "b607451c74364485a977b236ec946af4"
format_stamp: "Formatted at 2026-06-20 02:22:03 on dist-test-slave-9vg1"
server_key: "7f216e00f164c416aa4392833bb316f2"
server_key_iv: "bb488feafbc43eab7b90789583f5f427"
server_key_version: "encryptionkey@0"
I20260620 02:22:03.248692 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.006s
I20260620 02:22:03.254537 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:03.256752 17751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:22:03.256999 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "b607451c74364485a977b236ec946af4"
format_stamp: "Formatted at 2026-06-20 02:22:03 on dist-test-slave-9vg1"
server_key: "7f216e00f164c416aa4392833bb316f2"
server_key_iv: "bb488feafbc43eab7b90789583f5f427"
server_key_version: "encryptionkey@0"
I20260620 02:22:03.257318 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:03.279556 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:03.284500 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:03.284781 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:03.310104 17751 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:39019
I20260620 02:22:03.310118 17827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:39019 every 8 connection(s)
I20260620 02:22:03.311971 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:22:03.313445 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 17751
I20260620 02:22:03.313714 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:22:03.314525 10141 external_mini_cluster.cc:1428] Setting key 550b442adb4eee3c8069b8a911993cd8
I20260620 02:22:03.317648 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:22:03 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:22:03.332244 17828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Bootstrap starting.
I20260620 02:22:03.335978 17828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:03.337220 17828 log.cc:824] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:03.340718 17828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: No bootstrap required, opened a new log
I20260620 02:22:03.340973 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:03.316934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.9.231.126:39019)
0620 02:22:03.317862 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:03.317888 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:03.317933 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:22:03.320046 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:03.320077 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:03.320628 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:03.321094 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:03.323185 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.324910 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.326771 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.327646 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.327799 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:03.333109 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:03.333119 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:03.333137 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:03.333187 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:03.336529 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:03.337360 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:03.337368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:03.337394 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:03.337592 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:03.338010 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:03.338025 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:03.338028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:03.338617 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:03.338936 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:03.339447 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 02:22:03.339905 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":448,"threads_started":1}
I20260620 02:22:03.350195 17828 raft_consensus.cc:348] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b607451c74364485a977b236ec946af4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39019 } }
I20260620 02:22:03.350742 17828 raft_consensus.cc:374] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:03.350872 17828 raft_consensus.cc:729] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b607451c74364485a977b236ec946af4, State: Initialized, Role: FOLLOWER
I20260620 02:22:03.351613 17828 consensus_queue.cc:260] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b607451c74364485a977b236ec946af4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39019 } }
I20260620 02:22:03.351922 17828 raft_consensus.cc:388] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:22:03.352113 17828 raft_consensus.cc:482] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:22:03.352315 17828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:03.355374 17828 raft_consensus.cc:504] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b607451c74364485a977b236ec946af4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39019 } }
I20260620 02:22:03.356003 17828 leader_election.cc:302] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b607451c74364485a977b236ec946af4; no voters:
I20260620 02:22:03.356729 17828 leader_election.cc:288] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:22:03.356882 17833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:03.357697 17833 raft_consensus.cc:686] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [term 1 LEADER]: Becoming Leader. State: Replica: b607451c74364485a977b236ec946af4, State: Running, Role: LEADER
I20260620 02:22:03.358359 17833 consensus_queue.cc:237] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b607451c74364485a977b236ec946af4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39019 } }
I20260620 02:22:03.359308 17828 sys_catalog.cc:564] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:22:03.361362 17835 sys_catalog.cc:455] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b607451c74364485a977b236ec946af4. Latest consensus state: current_term: 1 leader_uuid: "b607451c74364485a977b236ec946af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b607451c74364485a977b236ec946af4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39019 } } }
I20260620 02:22:03.361900 17835 sys_catalog.cc:458] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:03.361372 17834 sys_catalog.cc:455] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b607451c74364485a977b236ec946af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b607451c74364485a977b236ec946af4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39019 } } }
I20260620 02:22:03.363356 17834 sys_catalog.cc:458] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:03.365789 17839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:22:03.372025 17839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:22:03.382606 17839 catalog_manager.cc:1347] Generated new cluster ID: 9d58a98f0f954cea850883fb10852ccd
I20260620 02:22:03.382774 17839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:22:03.400537 17839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:22:03.402335 17839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:22:03.416476 17839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Generated new TSK 0
I20260620 02:22:03.417862 17839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab.
Entry for principal oryx/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab.
I20260620 02:22:03.488517 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39019
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab
--principal=oryx/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:03.854686 17854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:03.855031 17854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:03.855211 17854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:03.863360 17854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:03.863497 17854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:03.863610 17854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:22:03.863726 17854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:22:03.874737 17854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab
--principal=oryx/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:03.876176 17854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:03.877550 17854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:03.893015 17860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:03.893111 17859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:03.893095 17862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:03.893774 17854 server_base.cc:1034] running on GCE node
I20260620 02:22:03.894516 17854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:03.895584 17854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:03.896828 17854 hybrid_clock.cc:648] HybridClock initialized: now 1781922123896762 us; error 42 us; skew 500 ppm
I20260620 02:22:03.897235 17854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:03 dist-test-slave-9vg1 krb5kdc[17738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922123, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:03.900792 17854 init.cc:377] Logged in from keytab as oryx/127.9.231.65@KRBTEST.COM (short username oryx)
I20260620 02:22:03.902671 17854 webserver.cc:460] Webserver started at http://127.9.231.65:36079/ using document root <none> and password file <none>
I20260620 02:22:03.903563 17854 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:03.903661 17854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:03.904002 17854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:03.907398 17854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4"
format_stamp: "Formatted at 2026-06-20 02:22:03 on dist-test-slave-9vg1"
server_key: "83fe055749f1151146c738272c27b045"
server_key_iv: "b3c0b70393ad9aebc4857cdf0cbf35fa"
server_key_version: "encryptionkey@0"
I20260620 02:22:03.908353 17854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4"
format_stamp: "Formatted at 2026-06-20 02:22:03 on dist-test-slave-9vg1"
server_key: "83fe055749f1151146c738272c27b045"
server_key_iv: "b3c0b70393ad9aebc4857cdf0cbf35fa"
server_key_version: "encryptionkey@0"
I20260620 02:22:03.914238 17854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:22:03.918344 17869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:03.920423 17854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:22:03.920637 17854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4"
format_stamp: "Formatted at 2026-06-20 02:22:03 on dist-test-slave-9vg1"
server_key: "83fe055749f1151146c738272c27b045"
server_key_iv: "b3c0b70393ad9aebc4857cdf0cbf35fa"
server_key_version: "encryptionkey@0"
I20260620 02:22:03.920900 17854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:03.934175 17854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:03.939004 17854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:03.939394 17854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:03.942322 17854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:03.942417 17854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:03.942615 17854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:03.942673 17854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:22:03 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922123, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:22:03.971848 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:03.943074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:43169 (local address 127.9.231.126:39019)
0620 02:22:03.944986 (+ 1912us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:03.944991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:03.945473 (+ 482us) server_negotiation.cc:407] Connection header received
0620 02:22:03.947894 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:03.947903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:03.948017 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:03.948148 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:03.950188 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.951553 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.954524 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.955308 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.955573 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:03.962229 (+ 6656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:03.962240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:03.962246 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:03.962303 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:03.966344 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:03.967867 (+ 1523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:03.967889 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:03.967896 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:03.968104 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:03.968878 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:03.968886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:03.968891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:03.969407 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:03.969575 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:03.970970 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0620 02:22:03.971361 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1772,"thread_start_us":1672,"threads_started":1}
I20260620 02:22:03.974257 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:03.944092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39019 (local address 127.9.231.65:43169)
0620 02:22:03.945245 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:03.945297 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:03.947527 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:03.948448 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:03.948470 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:03.949287 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:03.949954 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:03.949974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.951776 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.951783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:03.954264 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:03.954283 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.955691 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.955702 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:03.956154 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:03.957178 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:03.957229 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:03.961833 (+ 4604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:03.966607 (+ 4774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:03.966625 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:03.966655 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:03.967646 (+ 991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:03.968284 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:03.968292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:03.968298 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:03.968705 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:03.969604 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:03.969618 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:03.970347 (+ 729us) client_negotiation.cc:769] Sending connection context
0620 02:22:03.970780 (+ 433us) client_negotiation.cc:240] Negotiation successful
0620 02:22:03.971658 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":573,"threads_started":1}
I20260620 02:22:03.989547 17854 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:35185
I20260620 02:22:03.989586 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:35185 every 8 connection(s)
I20260620 02:22:03.991261 17854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:22:03.993747 17991 heartbeater.cc:347] Connected to a master server at 127.9.231.126:39019
I20260620 02:22:03.994052 17991 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:03.994994 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 17854
I20260620 02:22:03.995026 17991 heartbeater.cc:511] Master 127.9.231.126:39019 requested a full tablet report, sending...
I20260620 02:22:03.995283 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:22:03.996065 10141 external_mini_cluster.cc:1428] Setting key a9d42f7d63db3f3b6ced120d060d9a6f
I20260620 02:22:03.998024 17783 ts_manager.cc:194] Registered new tserver with Master: 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185)
I20260620 02:22:04.000038 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:03.982631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39019 (local address 127.9.231.65:35067)
0620 02:22:03.983544 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:03.983570 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:03.983836 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:03.984355 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:03.984359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:03.985165 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:03.985782 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:03.985799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.987578 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.987582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:03.989304 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:03.989326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.990437 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.990443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:03.990592 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:03.991419 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:03.991463 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:03.992657 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:03.997118 (+ 4461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:03.997124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:03.997127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:03.997573 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:03.998018 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:03.998024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:03.998028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:03.998220 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:03.998776 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:03.998781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:03.998995 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 02:22:03.999218 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 02:22:03.999587 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":441,"threads_started":1}
I20260620 02:22:04.000341 17783 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:43169
I20260620 02:22:04.000062 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:03.982673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:35067 (local address 127.9.231.126:39019)
0620 02:22:03.982935 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:03.982945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:03.983699 (+ 754us) server_negotiation.cc:407] Connection header received
0620 02:22:03.984021 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:03.984030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:03.984167 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:03.984305 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:03.986020 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.987372 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.989535 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:03.990188 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:03.990338 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:03.992981 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:03.992988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:03.992993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:03.993037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:03.996940 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:03.997719 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:03.997725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:03.997728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:03.997890 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:03.998319 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:03.998326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:03.998330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:03.998659 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:03.998836 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:03.999344 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 02:22:03.999610 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for oryx/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab.
Entry for principal oryx/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab.
I20260620 02:22:04.051378 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39019
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab
--principal=oryx/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:04.409047 17995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:04.409341 17995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:04.409511 17995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:04.417358 17995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:04.417532 17995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:04.417676 17995 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:22:04.417814 17995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:22:04.428459 17995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab
--principal=oryx/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:04.430073 17995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:04.431787 17995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:04.446776 18001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:04.446805 18004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:04.446724 18002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:04.447686 17995 server_base.cc:1034] running on GCE node
I20260620 02:22:04.448415 17995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:04.449594 17995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:04.450945 17995 hybrid_clock.cc:648] HybridClock initialized: now 1781922124450823 us; error 43 us; skew 500 ppm
I20260620 02:22:04.451370 17995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:04 dist-test-slave-9vg1 krb5kdc[17738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922124, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:04.455241 17995 init.cc:377] Logged in from keytab as oryx/127.9.231.66@KRBTEST.COM (short username oryx)
I20260620 02:22:04.457321 17995 webserver.cc:460] Webserver started at http://127.9.231.66:42445/ using document root <none> and password file <none>
I20260620 02:22:04.458197 17995 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:04.458361 17995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:04.458752 17995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:04.462328 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "0d9c7aefb7d044b4af2c26ef7c09f096"
format_stamp: "Formatted at 2026-06-20 02:22:04 on dist-test-slave-9vg1"
server_key: "90082721df5619551096d0e9b5bc92f6"
server_key_iv: "9a4906e078aaeadf9914ff9a98b81292"
server_key_version: "encryptionkey@0"
I20260620 02:22:04.463456 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "0d9c7aefb7d044b4af2c26ef7c09f096"
format_stamp: "Formatted at 2026-06-20 02:22:04 on dist-test-slave-9vg1"
server_key: "90082721df5619551096d0e9b5bc92f6"
server_key_iv: "9a4906e078aaeadf9914ff9a98b81292"
server_key_version: "encryptionkey@0"
I20260620 02:22:04.469930 17995 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:22:04.474416 18011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:04.476377 17995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:22:04.476629 17995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "0d9c7aefb7d044b4af2c26ef7c09f096"
format_stamp: "Formatted at 2026-06-20 02:22:04 on dist-test-slave-9vg1"
server_key: "90082721df5619551096d0e9b5bc92f6"
server_key_iv: "9a4906e078aaeadf9914ff9a98b81292"
server_key_version: "encryptionkey@0"
I20260620 02:22:04.476930 17995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:04.490286 17995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:04.495657 17995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:04.496013 17995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:04.499328 17995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:04.499476 17995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:04.499639 17995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:04.499718 17995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:22:04 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922124, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:22:04.528519 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:04.500394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:60849 (local address 127.9.231.126:39019)
0620 02:22:04.500657 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:04.500665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:04.502158 (+ 1493us) server_negotiation.cc:407] Connection header received
0620 02:22:04.504741 (+ 2583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:04.504747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:04.504855 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:04.504986 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:04.507809 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.508980 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.511847 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.512524 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.512682 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:04.518387 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:04.518393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:04.518397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:04.518438 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:04.522478 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:04.524096 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:04.524107 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:04.524113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:04.524342 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:04.525251 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:04.525288 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:04.525292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:04.525886 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:04.526075 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:04.527540 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0620 02:22:04.527916 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:22:04.529157 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:04.501046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39019 (local address 127.9.231.66:60849)
0620 02:22:04.501947 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:04.501999 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:04.504370 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:04.505435 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:04.505452 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:04.506405 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:04.507516 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:04.507554 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.509204 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.509216 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:04.511656 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:04.511674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.512886 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.512896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:04.513325 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:04.514208 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:04.514256 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:04.517990 (+ 3734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:04.522705 (+ 4715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:04.522724 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:04.522754 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:04.523871 (+ 1117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:04.524546 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:04.524555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:04.524561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:04.525038 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:04.526213 (+ 1175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:04.526221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:04.526866 (+ 645us) client_negotiation.cc:769] Sending connection context
0620 02:22:04.527312 (+ 446us) client_negotiation.cc:240] Negotiation successful
0620 02:22:04.527823 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":379,"threads_started":1}
I20260620 02:22:04.555399 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:04.536677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:47169 (local address 127.9.231.126:39019)
0620 02:22:04.536951 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:04.536961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:04.537073 (+ 112us) server_negotiation.cc:407] Connection header received
0620 02:22:04.537287 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:04.537296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:04.537435 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:04.537587 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:04.538797 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.540225 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.542573 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.543507 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.543692 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:04.547361 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:04.547367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:04.547371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:04.547421 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:04.551393 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:04.552510 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:04.552518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:04.552523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:04.552747 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:04.553405 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:04.553412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:04.553417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:04.553943 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:04.554077 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:04.554832 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 02:22:04.555124 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:22:04.555925 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:04.536121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39019 (local address 127.9.231.66:47169)
0620 02:22:04.536941 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:04.536969 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:04.537145 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:04.537640 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:04.537643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:04.538138 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:04.538590 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:04.538608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.540541 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.540546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:04.542243 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:04.542261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:04.544046 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:04.544057 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:04.544327 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:04.545371 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:04.545395 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:04.546929 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:04.551592 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:04.551601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:04.551608 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:04.552312 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:04.552945 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:04.552954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:04.552960 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:04.553192 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:04.554183 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:04.554191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:04.554448 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 02:22:04.554680 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 02:22:04.555203 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"spinlock_wait_cycles":22400,"thread_start_us":587,"threads_started":1}
I20260620 02:22:04.563791 17995 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:43155
I20260620 02:22:04.563809 18130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:43155 every 8 connection(s)
I20260620 02:22:04.565317 17995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:22:04.566627 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 17995
I20260620 02:22:04.566859 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:22:04.567425 10141 external_mini_cluster.cc:1428] Setting key ba220d0bf57c337f3abcfac39f96b8dc
I20260620 02:22:04.568073 18131 heartbeater.cc:347] Connected to a master server at 127.9.231.126:39019
I20260620 02:22:04.568352 18131 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:04.569283 18131 heartbeater.cc:511] Master 127.9.231.126:39019 requested a full tablet report, sending...
I20260620 02:22:04.570907 17782 ts_manager.cc:194] Registered new tserver with Master: 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155)
I20260620 02:22:04.572309 17782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:60849
WARNING: no policy specified for oryx/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab.
Entry for principal oryx/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab.
I20260620 02:22:04.622713 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39019
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab
--principal=oryx/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:04.987026 18135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:04.987438 18135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:04.987552 18135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:04.996920 18135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:04.997144 18135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:04.997239 18135 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:22:04.997351 18135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:22:05.004894 17991 heartbeater.cc:503] Master 127.9.231.126:39019 was elected leader, sending a full tablet report...
I20260620 02:22:05.009366 18135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab
--principal=oryx/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.231.126:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:05.010937 18135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:05.012517 18135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:05.028997 18144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:05.029023 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:05.029138 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:05.029763 18135 server_base.cc:1034] running on GCE node
I20260620 02:22:05.030751 18135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:05.032338 18135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:05.033636 18135 hybrid_clock.cc:648] HybridClock initialized: now 1781922125033570 us; error 49 us; skew 500 ppm
I20260620 02:22:05.034008 18135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:05 dist-test-slave-9vg1 krb5kdc[17738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922125, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:05.037853 18135 init.cc:377] Logged in from keytab as oryx/127.9.231.67@KRBTEST.COM (short username oryx)
I20260620 02:22:05.039734 18135 webserver.cc:460] Webserver started at http://127.9.231.67:39557/ using document root <none> and password file <none>
I20260620 02:22:05.040529 18135 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:05.040632 18135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:05.041055 18135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:05.044426 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "8a1596d961cd4d3cbfe77307ec36b02a"
format_stamp: "Formatted at 2026-06-20 02:22:05 on dist-test-slave-9vg1"
server_key: "1464c67e2be56f689bb6b302b51432fb"
server_key_iv: "f9e404d04be1dc0081d6463a056cdb26"
server_key_version: "encryptionkey@0"
I20260620 02:22:05.045286 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "8a1596d961cd4d3cbfe77307ec36b02a"
format_stamp: "Formatted at 2026-06-20 02:22:05 on dist-test-slave-9vg1"
server_key: "1464c67e2be56f689bb6b302b51432fb"
server_key_iv: "f9e404d04be1dc0081d6463a056cdb26"
server_key_version: "encryptionkey@0"
I20260620 02:22:05.051383 18135 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:22:05.055840 18151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:05.057848 18135 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:22:05.058030 18135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "8a1596d961cd4d3cbfe77307ec36b02a"
format_stamp: "Formatted at 2026-06-20 02:22:05 on dist-test-slave-9vg1"
server_key: "1464c67e2be56f689bb6b302b51432fb"
server_key_iv: "f9e404d04be1dc0081d6463a056cdb26"
server_key_version: "encryptionkey@0"
I20260620 02:22:05.058320 18135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:05.088621 18135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:05.093581 18135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:05.093904 18135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:05.098104 18135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:05.098248 18135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:05.098479 18135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:05.098668 18135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:22:05 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922125, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:22:05.129042 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:05.098042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:35683 (local address 127.9.231.126:39019)
0620 02:22:05.098837 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:05.098844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:05.100192 (+ 1348us) server_negotiation.cc:407] Connection header received
0620 02:22:05.102429 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:05.102434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:05.102550 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:05.102681 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:05.105393 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.106819 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.110077 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.110544 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.110704 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:05.116637 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:05.116648 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:05.116654 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:05.116710 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:05.123311 (+ 6601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:05.124965 (+ 1654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:05.124972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:05.124976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:05.125187 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:05.126010 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:05.126018 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:05.126021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:05.126488 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:05.126699 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:05.128210 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0620 02:22:05.128540 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":494,"threads_started":1}
I20260620 02:22:05.130836 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:05.098792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39019 (local address 127.9.231.67:35683)
0620 02:22:05.099950 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:05.100017 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:05.102081 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:05.102969 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:05.102979 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:05.103849 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:05.105076 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:05.105122 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.107106 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.107115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:05.109842 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:05.109866 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.110826 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.110835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:05.111241 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:05.112769 (+ 1528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:05.112796 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:05.116150 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:05.123569 (+ 7419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:05.123583 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:05.123616 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:05.124708 (+ 1092us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:05.125388 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:05.125400 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:05.125406 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:05.125844 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:05.126799 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:05.126814 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:05.127546 (+ 732us) client_negotiation.cc:769] Sending connection context
0620 02:22:05.128041 (+ 495us) client_negotiation.cc:240] Negotiation successful
0620 02:22:05.128812 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":586,"threads_started":1}
I20260620 02:22:05.148186 18135 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:46219
I20260620 02:22:05.148289 18271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:46219 every 8 connection(s)
I20260620 02:22:05.149811 18135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:22:05.152385 18272 heartbeater.cc:347] Connected to a master server at 127.9.231.126:39019
I20260620 02:22:05.152673 18272 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:05.153393 18272 heartbeater.cc:511] Master 127.9.231.126:39019 requested a full tablet report, sending...
I20260620 02:22:05.154917 17782 ts_manager.cc:194] Registered new tserver with Master: 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:05.155817 17782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:35683
I20260620 02:22:05.156510 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:05.140021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:53527 (local address 127.9.231.126:39019)
0620 02:22:05.140248 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:05.140255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:05.141200 (+ 945us) server_negotiation.cc:407] Connection header received
0620 02:22:05.141503 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:05.141509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:05.141601 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:05.141727 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:05.143689 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.144828 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.146766 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.147430 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.147582 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:05.150094 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:05.150099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:05.150102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:05.150140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:05.153474 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:05.154244 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:05.154248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:05.154251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:05.154431 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:05.154830 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:05.154836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:05.154839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:05.155271 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:05.155448 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:05.155858 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 02:22:05.156136 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:22:05.156759 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:05.140309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39019 (local address 127.9.231.67:53527)
0620 02:22:05.141046 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:05.141072 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:05.141348 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:05.141968 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:05.141974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:05.142904 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:05.143502 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:05.143522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.145042 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.145046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:05.146581 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:05.146598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.147678 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.147685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:05.148007 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:05.148815 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:05.148833 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:05.149797 (+ 964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:05.153653 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:05.153659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:05.153663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:05.154106 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:05.154568 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:05.154572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:05.154574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:05.154704 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:05.155409 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:05.155415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:05.155607 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 02:22:05.155758 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 02:22:05.156163 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":408,"threads_started":1}
I20260620 02:22:05.159780 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 18135
I20260620 02:22:05.159988 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:22:05.160573 10141 external_mini_cluster.cc:1428] Setting key 3e4eec5401cf4542b19c99289f3e18d1
I20260620 02:22:05.164346 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:22:05 dist-test-slave-9vg1 krb5kdc[17738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:22:05 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:05 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:05.575814 18131 heartbeater.cc:503] Master 127.9.231.126:39019 was elected leader, sending a full tablet report...
I20260620 02:22:06.159274 18272 heartbeater.cc:503] Master 127.9.231.126:39019 was elected leader, sending a full tablet report...
W20260620 02:22:06.301965 18281 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.088s user 0.001s sys 0.015s
W20260620 02:22:06.303424 18159 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:05.207864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.9.231.126:39019)
0620 02:22:05.208125 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:05.208133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:05.208464 (+ 331us) server_negotiation.cc:407] Connection header received
0620 02:22:05.208641 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:05.208646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:05.208720 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:05.208819 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:05.210495 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.211379 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.212884 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:05.213321 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:05.213456 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.303016 (+1089560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43652: BlockingRecv error: recv got EOF from 127.0.0.1:43652 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 02:22:06 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:22:06.331416 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.307587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.9.231.126:39019)
0620 02:22:06.307955 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.307962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.308259 (+ 297us) server_negotiation.cc:407] Connection header received
0620 02:22:06.308546 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.308552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.308620 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.308737 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:06.310075 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.310817 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.312206 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.312693 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.312851 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.324943 (+ 12092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:06.324955 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:06.324959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:06.325007 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:06.328289 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:06.329234 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:06.329240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:06.329245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:06.329401 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:06.329826 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:06.329831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:06.329834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:06.330137 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:06.330409 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.330771 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.331029 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:22:06.336434 17782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:22:06.343912 17782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:22:06.369659 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.357293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35185 (local address 127.0.0.1:52222)
0620 02:22:06.358416 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.358482 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.358914 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.360523 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.360537 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.360587 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:06.361140 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.361158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.363518 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.363524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.365924 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.365939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.368537 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.368546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.368802 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.368880 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.369023 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.369250 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":471,"threads_started":1}
I20260620 02:22:06.370779 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.357527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.9.231.65:35185)
0620 02:22:06.359673 (+ 2146us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.359685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.359720 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:22:06.359882 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.359891 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.360198 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.361437 (+ 1239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:06.361680 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.363381 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.366114 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.367787 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.370014 (+ 2227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.370256 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.370324 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.370473 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1964,"thread_start_us":1802,"threads_started":1}
I20260620 02:22:06.371892 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.358883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:43155 (local address 127.0.0.1:41696)
0620 02:22:06.361357 (+ 2474us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.361378 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.361598 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.362307 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.362311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.362337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:06.362844 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.362860 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.365148 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.365153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.368303 (+ 3150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.368326 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.370937 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.370945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.371202 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.371241 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.371350 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.371505 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2336,"mutex_wait_us":199,"thread_start_us":1785,"threads_started":1}
I20260620 02:22:06.371956 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.359271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.9.231.66:43155)
0620 02:22:06.360009 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.360016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.361478 (+ 1462us) server_negotiation.cc:407] Connection header received
0620 02:22:06.361797 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.361806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.362134 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.362315 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:06.363182 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.364931 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.368572 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.370319 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.370466 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.370691 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.371437 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.371651 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":380,"threads_started":1}
I20260620 02:22:06.372138 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.358444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.9.231.67:46219)
0620 02:22:06.360470 (+ 2026us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.360478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.362058 (+ 1580us) server_negotiation.cc:407] Connection header received
0620 02:22:06.362295 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.362303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.362558 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.362770 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:06.363449 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.365122 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.368584 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.370494 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.370660 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.370871 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.371638 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.371831 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1850,"thread_start_us":1725,"threads_started":1}
I20260620 02:22:06.372668 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.357830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:46219 (local address 127.0.0.1:58186)
0620 02:22:06.361770 (+ 3940us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.361789 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.361904 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.362776 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.362781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.362810 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:06.363244 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.363256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.365293 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.365297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.368303 (+ 3006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.368326 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.370745 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.370751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.371025 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.371093 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.371187 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.371350 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2987,"mutex_wait_us":149,"spinlock_wait_cycles":867712,"thread_start_us":3237,"threads_started":1}
I20260620 02:22:06.375536 17921 tablet_service.cc:1467] Processing CreateTablet for tablet d351cfaf3d2649b190e3b84b339dab32 (DEFAULT_TABLE table=test-table [id=cdde685b9e754e0997ad126890b0ee10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:06.376116 18061 tablet_service.cc:1467] Processing CreateTablet for tablet d351cfaf3d2649b190e3b84b339dab32 (DEFAULT_TABLE table=test-table [id=cdde685b9e754e0997ad126890b0ee10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:06.376480 18202 tablet_service.cc:1467] Processing CreateTablet for tablet d351cfaf3d2649b190e3b84b339dab32 (DEFAULT_TABLE table=test-table [id=cdde685b9e754e0997ad126890b0ee10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:06.376963 17921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d351cfaf3d2649b190e3b84b339dab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.377534 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d351cfaf3d2649b190e3b84b339dab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.377804 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d351cfaf3d2649b190e3b84b339dab32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.390857 18298 tablet_bootstrap.cc:492] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Bootstrap starting.
I20260620 02:22:06.395700 18298 tablet_bootstrap.cc:654] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.395898 18299 tablet_bootstrap.cc:492] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Bootstrap starting.
I20260620 02:22:06.397904 18298 log.cc:824] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:06.399618 18300 tablet_bootstrap.cc:492] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Bootstrap starting.
I20260620 02:22:06.400105 18299 tablet_bootstrap.cc:654] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.401553 18299 log.cc:824] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:06.404300 18300 tablet_bootstrap.cc:654] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.405525 18299 tablet_bootstrap.cc:492] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: No bootstrap required, opened a new log
I20260620 02:22:06.405551 18298 tablet_bootstrap.cc:492] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: No bootstrap required, opened a new log
I20260620 02:22:06.405853 18299 ts_tablet_manager.cc:1397] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:22:06.405872 18298 ts_tablet_manager.cc:1397] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260620 02:22:06.405884 18300 log.cc:824] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:06.410044 18300 tablet_bootstrap.cc:492] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: No bootstrap required, opened a new log
I20260620 02:22:06.410423 18300 ts_tablet_manager.cc:1397] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 02:22:06.415519 18298 raft_consensus.cc:348] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.416124 18298 raft_consensus.cc:374] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.416314 18298 raft_consensus.cc:729] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142a6374f0d0478d8ca4c73c0a8ff2c4, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.417085 18298 consensus_queue.cc:260] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.417250 18299 raft_consensus.cc:348] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.417829 18299 raft_consensus.cc:374] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.417996 18299 raft_consensus.cc:729] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9c7aefb7d044b4af2c26ef7c09f096, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.418762 18299 consensus_queue.cc:260] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.419301 18298 ts_tablet_manager.cc:1428] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260620 02:22:06.424382 18299 ts_tablet_manager.cc:1428] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent starting tablet: real 0.018s user 0.006s sys 0.013s
I20260620 02:22:06.425716 18300 raft_consensus.cc:348] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.426318 18300 raft_consensus.cc:374] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.426438 18300 raft_consensus.cc:729] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1596d961cd4d3cbfe77307ec36b02a, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.427224 18300 consensus_queue.cc:260] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.429277 18300 ts_tablet_manager.cc:1428] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
W20260620 02:22:06.496367 17992 tablet.cc:2369] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:22:06.525818 18304 raft_consensus.cc:482] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:22:06.526222 18304 raft_consensus.cc:504] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.528223 18304 leader_election.cc:288] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:06.536187 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.528793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:51607 (local address 127.9.231.67:46219)
0620 02:22:06.529000 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.529008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.529606 (+ 598us) server_negotiation.cc:407] Connection header received
0620 02:22:06.529848 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.529853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.529935 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.530059 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:06.530792 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.531910 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.534154 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.535023 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.535287 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.535385 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.535763 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.535905 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:22:06.536192 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.528693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:41015 (local address 127.9.231.66:43155)
0620 02:22:06.528942 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.528951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.529657 (+ 706us) server_negotiation.cc:407] Connection header received
0620 02:22:06.529862 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.529866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.529939 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.530056 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:06.530793 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.531712 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.534084 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.535033 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.535323 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.535414 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.535813 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.535945 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:22:06.536386 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.528682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:43155 (local address 127.9.231.65:41015)
0620 02:22:06.529493 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.529510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.529691 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.530080 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.530084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.530112 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:06.530607 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.530624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.531850 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.531858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.533870 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.533884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.535399 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.535408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.535609 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.535634 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.535758 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.535901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":189,"thread_start_us":483,"threads_started":1}
I20260620 02:22:06.536360 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.528526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:46219 (local address 127.9.231.65:51607)
0620 02:22:06.529458 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.529485 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.529661 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.530091 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.530096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.530122 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:06.530631 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.530641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.532090 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.532094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.533856 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.533872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.535355 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.535362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.535542 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.535605 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.535691 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.535863 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":15,"thread_start_us":511,"threads_started":1}
I20260620 02:22:06.537796 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d351cfaf3d2649b190e3b84b339dab32" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" is_pre_election: true
I20260620 02:22:06.537796 18085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d351cfaf3d2649b190e3b84b339dab32" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" is_pre_election: true
I20260620 02:22:06.538441 18085 raft_consensus.cc:2454] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 0.
I20260620 02:22:06.538446 18222 raft_consensus.cc:2454] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 0.
I20260620 02:22:06.539300 17870 leader_election.cc:302] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d9c7aefb7d044b4af2c26ef7c09f096, 142a6374f0d0478d8ca4c73c0a8ff2c4, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:06.539908 18304 raft_consensus.cc:2781] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:22:06.540120 18304 raft_consensus.cc:482] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:22:06.540243 18304 raft_consensus.cc:3037] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:06.542634 18304 raft_consensus.cc:504] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.543435 18304 leader_election.cc:288] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:06.544106 18085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d351cfaf3d2649b190e3b84b339dab32" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096"
I20260620 02:22:06.544127 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d351cfaf3d2649b190e3b84b339dab32" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a"
I20260620 02:22:06.544353 18085 raft_consensus.cc:3037] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:06.544395 18222 raft_consensus.cc:3037] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:06.546741 18222 raft_consensus.cc:2454] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 1.
I20260620 02:22:06.546741 18085 raft_consensus.cc:2454] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 1.
I20260620 02:22:06.547341 17870 leader_election.cc:302] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d9c7aefb7d044b4af2c26ef7c09f096, 142a6374f0d0478d8ca4c73c0a8ff2c4, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:06.547724 18304 raft_consensus.cc:2781] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:06.548267 18304 raft_consensus.cc:686] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 LEADER]: Becoming Leader. State: Replica: 142a6374f0d0478d8ca4c73c0a8ff2c4, State: Running, Role: LEADER
I20260620 02:22:06.548813 18304 consensus_queue.cc:237] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.554242 17782 catalog_manager.cc:5495] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65). New cstate: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } health_report { overall_health: UNKNOWN } } }
W20260620 02:22:06.569103 18132 tablet.cc:2369] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:22:06.572485 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.566393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.9.231.65:35185)
0620 02:22:06.566633 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.566638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.566657 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:22:06.566992 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.566997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.567177 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.567323 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:06.567745 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.568604 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.570390 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.570781 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.570881 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.571354 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:06.572014 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:06.572123 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.572195 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.572266 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 02:22:06.581944 18085 raft_consensus.cc:1261] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Refusing update from remote peer 142a6374f0d0478d8ca4c73c0a8ff2c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:06.581940 18222 raft_consensus.cc:1261] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Refusing update from remote peer 142a6374f0d0478d8ca4c73c0a8ff2c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:06.583220 18304 consensus_queue.cc:1035] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:06.583630 18309 consensus_queue.cc:1035] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:06.600339 18314 mvcc.cc:205] Tried to move back new op lower bound from 7298753030467289088 to 7298753030348300288. Current Snapshot: MvccSnapshot[applied={T|T < 7298753030467289088}]
I20260620 02:22:06.613612 17782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:06.613927 17782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:06.618259 17782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Sending DeleteTablet for 3 replicas of tablet d351cfaf3d2649b190e3b84b339dab32
I20260620 02:22:06.620040 18202 tablet_service.cc:1514] Processing DeleteTablet for tablet d351cfaf3d2649b190e3b84b339dab32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:06 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:58186
I20260620 02:22:06.620039 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet d351cfaf3d2649b190e3b84b339dab32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:06 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:41696
I20260620 02:22:06.620873 17921 tablet_service.cc:1514] Processing DeleteTablet for tablet d351cfaf3d2649b190e3b84b339dab32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:06 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:52222
I20260620 02:22:06.621188 18323 tablet_replica.cc:330] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: stopping tablet replica
I20260620 02:22:06.621698 18323 raft_consensus.cc:2229] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:06.622267 18323 raft_consensus.cc:2258] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:06.622262 17782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:22:06.623181 18322 tablet_replica.cc:330] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: stopping tablet replica
I20260620 02:22:06.623430 18324 tablet_replica.cc:330] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: stopping tablet replica
I20260620 02:22:06.623677 18322 raft_consensus.cc:2229] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 02:22:06.623904 17782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:22:06.623925 18324 raft_consensus.cc:2229] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:22:06.624293 18322 raft_consensus.cc:2258] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:06.624899 18324 raft_consensus.cc:2258] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:06.627202 18322 ts_tablet_manager.cc:1905] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:06.627205 18324 ts_tablet_manager.cc:1905] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:06.634514 18202 tablet_service.cc:1467] Processing CreateTablet for tablet 506bea39fd2e4f0889df140eb082d9d2 (DEFAULT_TABLE table=test-table [id=4b380676c77d469392ed13f5dff2f76e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:06.635159 18322 ts_tablet_manager.cc:1918] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:06.635320 18324 ts_tablet_manager.cc:1918] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:06.635449 18322 log.cc:1196] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/d351cfaf3d2649b190e3b84b339dab32
I20260620 02:22:06.635653 18324 log.cc:1196] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/d351cfaf3d2649b190e3b84b339dab32
I20260620 02:22:06.636119 18322 ts_tablet_manager.cc:1939] T d351cfaf3d2649b190e3b84b339dab32 P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting consensus metadata
I20260620 02:22:06.636438 18324 ts_tablet_manager.cc:1939] T d351cfaf3d2649b190e3b84b339dab32 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting consensus metadata
I20260620 02:22:06.636392 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 506bea39fd2e4f0889df140eb082d9d2 (DEFAULT_TABLE table=test-table [id=4b380676c77d469392ed13f5dff2f76e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:06.636911 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506bea39fd2e4f0889df140eb082d9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.638315 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506bea39fd2e4f0889df140eb082d9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.638590 17769 catalog_manager.cc:4841] TS 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185): tablet d351cfaf3d2649b190e3b84b339dab32 (table test-table [id=cdde685b9e754e0997ad126890b0ee10]) successfully deleted
I20260620 02:22:06.639500 17767 catalog_manager.cc:4841] TS 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219): tablet d351cfaf3d2649b190e3b84b339dab32 (table test-table [id=cdde685b9e754e0997ad126890b0ee10]) successfully deleted
I20260620 02:22:06.639492 17921 tablet_service.cc:1467] Processing CreateTablet for tablet 506bea39fd2e4f0889df140eb082d9d2 (DEFAULT_TABLE table=test-table [id=4b380676c77d469392ed13f5dff2f76e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:06.639978 17921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506bea39fd2e4f0889df140eb082d9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.642493 18323 ts_tablet_manager.cc:1905] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:06.644840 18299 tablet_bootstrap.cc:492] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: Bootstrap starting.
I20260620 02:22:06.645298 18300 tablet_bootstrap.cc:492] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Bootstrap starting.
I20260620 02:22:06.646847 18299 tablet_bootstrap.cc:654] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.646934 18300 tablet_bootstrap.cc:654] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.648268 18298 tablet_bootstrap.cc:492] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Bootstrap starting.
I20260620 02:22:06.650063 18298 tablet_bootstrap.cc:654] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.650388 18323 ts_tablet_manager.cc:1918] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:06.650579 18323 log.cc:1196] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/d351cfaf3d2649b190e3b84b339dab32
I20260620 02:22:06.651212 18323 ts_tablet_manager.cc:1939] T d351cfaf3d2649b190e3b84b339dab32 P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting consensus metadata
I20260620 02:22:06.653477 17769 catalog_manager.cc:4841] TS 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155): tablet d351cfaf3d2649b190e3b84b339dab32 (table test-table [id=cdde685b9e754e0997ad126890b0ee10]) successfully deleted
I20260620 02:22:06.659895 18300 tablet_bootstrap.cc:492] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: No bootstrap required, opened a new log
I20260620 02:22:06.660226 18300 ts_tablet_manager.cc:1397] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260620 02:22:06.661109 18298 tablet_bootstrap.cc:492] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: No bootstrap required, opened a new log
I20260620 02:22:06.661218 18300 raft_consensus.cc:348] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.661402 18298 ts_tablet_manager.cc:1397] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260620 02:22:06.661526 18300 raft_consensus.cc:374] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.661672 18300 raft_consensus.cc:729] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1596d961cd4d3cbfe77307ec36b02a, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.662063 18300 consensus_queue.cc:260] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.662840 18300 ts_tablet_manager.cc:1428] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:22:06.663872 18299 tablet_bootstrap.cc:492] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: No bootstrap required, opened a new log
I20260620 02:22:06.663764 18298 raft_consensus.cc:348] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.664058 18298 raft_consensus.cc:374] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.664163 18299 ts_tablet_manager.cc:1397] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.006s
I20260620 02:22:06.664191 18298 raft_consensus.cc:729] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142a6374f0d0478d8ca4c73c0a8ff2c4, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.664661 18298 consensus_queue.cc:260] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.665107 18299 raft_consensus.cc:348] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.665359 18299 raft_consensus.cc:374] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.665470 18298 ts_tablet_manager.cc:1428] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 02:22:06.665477 18299 raft_consensus.cc:729] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9c7aefb7d044b4af2c26ef7c09f096, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.665810 18299 consensus_queue.cc:260] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.666496 18299 ts_tablet_manager.cc:1428] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 02:22:06.821816 18305 raft_consensus.cc:482] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:22:06.822075 18305 raft_consensus.cc:504] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.823908 18305 leader_election.cc:288] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:06.833737 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.824660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:59701 (local address 127.9.231.67:46219)
0620 02:22:06.824848 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.824857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.825560 (+ 703us) server_negotiation.cc:407] Connection header received
0620 02:22:06.826329 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.826335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.826422 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.826515 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:06.827123 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.827867 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.831195 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.832577 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.832781 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.832859 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.833216 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.833416 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:22:06.833824 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.824239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:46219 (local address 127.9.231.66:59701)
0620 02:22:06.825347 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.825444 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.825728 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.826567 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.826572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.826596 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:06.826950 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.826958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.828013 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.828020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.830927 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.830951 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.832835 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.832841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.832979 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.833008 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.833152 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.833329 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":167,"thread_start_us":1094,"threads_started":1}
I20260620 02:22:06.835284 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506bea39fd2e4f0889df140eb082d9d2" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" is_pre_election: true
I20260620 02:22:06.835392 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.824466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:57723 (local address 127.9.231.65:35185)
0620 02:22:06.824739 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.824749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.825786 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 02:22:06.825923 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.825929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.826014 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.826121 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:06.826789 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.828011 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.832967 (+ 4956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.834260 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.834436 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.834509 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.834926 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.835097 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:22:06.835490 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.824258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35185 (local address 127.9.231.66:57723)
0620 02:22:06.825678 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.825695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.825798 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.826166 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.826171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.826200 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:06.826578 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.826589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.828151 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.828157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.830927 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.830951 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.834492 (+ 3541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.834499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.834704 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.834748 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.834925 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.835114 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":142,"thread_start_us":736,"threads_started":1}
I20260620 02:22:06.835599 18222 raft_consensus.cc:2454] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 0.
I20260620 02:22:06.836336 18012 leader_election.cc:302] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9c7aefb7d044b4af2c26ef7c09f096, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:06.836426 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506bea39fd2e4f0889df140eb082d9d2" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" is_pre_election: true
I20260620 02:22:06.836798 18305 raft_consensus.cc:2781] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:22:06.836928 18305 raft_consensus.cc:482] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:22:06.836889 17941 raft_consensus.cc:2454] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 0.
I20260620 02:22:06.837039 18305 raft_consensus.cc:3037] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:06.839496 18305 raft_consensus.cc:504] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.840214 18305 leader_election.cc:288] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 election: Requested vote from peers 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:06.840826 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506bea39fd2e4f0889df140eb082d9d2" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4"
I20260620 02:22:06.840932 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506bea39fd2e4f0889df140eb082d9d2" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a"
I20260620 02:22:06.841085 17941 raft_consensus.cc:3037] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:06.841187 18222 raft_consensus.cc:3037] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:06.843921 17941 raft_consensus.cc:2454] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 1.
I20260620 02:22:06.844401 18222 raft_consensus.cc:2454] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 1.
I20260620 02:22:06.844599 18014 leader_election.cc:302] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9c7aefb7d044b4af2c26ef7c09f096, 142a6374f0d0478d8ca4c73c0a8ff2c4; no voters:
I20260620 02:22:06.845012 18305 raft_consensus.cc:2781] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:06.845541 18305 raft_consensus.cc:686] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9c7aefb7d044b4af2c26ef7c09f096, State: Running, Role: LEADER
I20260620 02:22:06.846328 18305 consensus_queue.cc:237] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.851423 17783 catalog_manager.cc:5495] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } health_report { overall_health: UNKNOWN } } }
W20260620 02:22:06.903566 18273 tablet.cc:2369] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:22:06 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922123, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.126@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:22:06.943641 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.926293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39019 (local address 127.0.0.1:43676)
0620 02:22:06.927149 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:06.927175 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:06.927365 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:06.927851 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:06.927861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:06.928913 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:06.929228 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.929246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.930535 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.930543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.932131 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:06.932152 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.933103 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.933110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:06.933327 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.934222 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:06.934247 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:06.937316 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:06.940767 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:06.940789 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:06.940798 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:06.941309 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:06.941715 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:06.941720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:06.941724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:06.941872 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:06.942450 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:06.942459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:06.942626 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 02:22:06.942796 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 02:22:06.943186 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":411,"threads_started":1}
I20260620 02:22:06.943661 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:06.926299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.9.231.126:39019)
0620 02:22:06.927499 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:06.927504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:06.927526 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:22:06.927584 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:06.927587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:06.927653 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:06.927750 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:06.929455 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.930336 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.932351 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:06.932868 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:06.932984 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:06.937611 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:06.937616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:06.937620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:06.937654 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:06.940545 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:06.941454 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:06.941465 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:06.941469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:06.941595 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:06.941995 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:06.942001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:06.942009 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:06.942305 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:06.942468 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:06.942903 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 02:22:06.943314 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":213,"server-negotiator.queue_time_us":1086,"spinlock_wait_cycles":17792,"thread_start_us":941,"threads_started":1}
I20260620 02:22:06.950459 17783 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:43676:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:22:06.951941 17783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:22:06.960493 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 41aae64e47f4485881c45aff5bd219aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d50b497b75cd44a39354976f3f3edb9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:22:06.960513 17921 tablet_service.cc:1467] Processing CreateTablet for tablet 41aae64e47f4485881c45aff5bd219aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d50b497b75cd44a39354976f3f3edb9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:22:06.961093 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aae64e47f4485881c45aff5bd219aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.961182 17921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aae64e47f4485881c45aff5bd219aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.967181 18299 tablet_bootstrap.cc:492] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: Bootstrap starting.
I20260620 02:22:06.968001 18202 tablet_service.cc:1467] Processing CreateTablet for tablet 41aae64e47f4485881c45aff5bd219aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d50b497b75cd44a39354976f3f3edb9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:22:06.968602 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aae64e47f4485881c45aff5bd219aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:06.968813 18299 tablet_bootstrap.cc:654] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.971027 18299 tablet_bootstrap.cc:492] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: No bootstrap required, opened a new log
I20260620 02:22:06.971271 18299 ts_tablet_manager.cc:1397] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:22:06.972033 18299 raft_consensus.cc:348] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.972275 18299 raft_consensus.cc:374] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.972364 18299 raft_consensus.cc:729] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9c7aefb7d044b4af2c26ef7c09f096, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.972844 18299 consensus_queue.cc:260] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.973528 18298 tablet_bootstrap.cc:492] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: Bootstrap starting.
I20260620 02:22:06.973419 18305 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:06.973740 18305 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:22:06.973747 18299 ts_tablet_manager.cc:1428] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:22:06.975623 18298 tablet_bootstrap.cc:654] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.978288 18300 tablet_bootstrap.cc:492] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: Bootstrap starting.
I20260620 02:22:06.980124 18300 tablet_bootstrap.cc:654] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:06.983208 18300 tablet_bootstrap.cc:492] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: No bootstrap required, opened a new log
I20260620 02:22:06.983402 18300 ts_tablet_manager.cc:1397] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 02:22:06.983634 18298 tablet_bootstrap.cc:492] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: No bootstrap required, opened a new log
I20260620 02:22:06.983817 18298 ts_tablet_manager.cc:1397] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 02:22:06.984252 18300 raft_consensus.cc:348] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.984537 18300 raft_consensus.cc:374] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.984659 18300 raft_consensus.cc:729] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1596d961cd4d3cbfe77307ec36b02a, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.984632 18298 raft_consensus.cc:348] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.984892 18298 raft_consensus.cc:374] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:06.984999 18298 raft_consensus.cc:729] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142a6374f0d0478d8ca4c73c0a8ff2c4, State: Initialized, Role: FOLLOWER
I20260620 02:22:06.984938 18300 consensus_queue.cc:260] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.985255 18298 consensus_queue.cc:260] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:06.985543 18300 ts_tablet_manager.cc:1428] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:22:06.985723 18309 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:06.986181 18309 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:22:06.986513 18306 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:06.986863 18306 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:22:06.988305 18298 ts_tablet_manager.cc:1428] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:22:07.203323 18309 raft_consensus.cc:482] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:22:07.203540 18309 raft_consensus.cc:504] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.204388 18309 leader_election.cc:288] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:07.205075 18085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aae64e47f4485881c45aff5bd219aa" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" is_pre_election: true
I20260620 02:22:07.205124 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aae64e47f4485881c45aff5bd219aa" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" is_pre_election: true
I20260620 02:22:07.205335 18085 raft_consensus.cc:2454] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 0.
I20260620 02:22:07.205397 18222 raft_consensus.cc:2454] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 0.
I20260620 02:22:07.205904 17870 leader_election.cc:302] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142a6374f0d0478d8ca4c73c0a8ff2c4, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:07.206212 18309 raft_consensus.cc:2781] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:22:07.206290 18309 raft_consensus.cc:482] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:22:07.206367 18309 raft_consensus.cc:3037] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:07.208273 18309 raft_consensus.cc:504] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.208931 18309 leader_election.cc:288] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:07.209460 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aae64e47f4485881c45aff5bd219aa" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a"
I20260620 02:22:07.209481 18085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aae64e47f4485881c45aff5bd219aa" candidate_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096"
I20260620 02:22:07.209758 18222 raft_consensus.cc:3037] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:07.209884 18085 raft_consensus.cc:3037] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:07.211931 18222 raft_consensus.cc:2454] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 1.
I20260620 02:22:07.212116 18085 raft_consensus.cc:2454] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142a6374f0d0478d8ca4c73c0a8ff2c4 in term 1.
I20260620 02:22:07.212553 17870 leader_election.cc:302] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142a6374f0d0478d8ca4c73c0a8ff2c4, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:07.212906 18309 raft_consensus.cc:2781] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:07.213387 18309 raft_consensus.cc:686] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 LEADER]: Becoming Leader. State: Replica: 142a6374f0d0478d8ca4c73c0a8ff2c4, State: Running, Role: LEADER
I20260620 02:22:07.213685 18309 consensus_queue.cc:237] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.215281 18342 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: TxnStatusTablet state changed. Reason: New leader 142a6374f0d0478d8ca4c73c0a8ff2c4. Latest consensus state: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:07.215600 18342 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:22:07.216189 18344 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:22:07.216420 18344 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:22:07.216902 17782 catalog_manager.cc:5495] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65). New cstate: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } health_report { overall_health: UNKNOWN } } }
I20260620 02:22:07.293699 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.286557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35185 (local address 127.0.0.1:52234)
0620 02:22:07.286851 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:07.286873 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:07.287335 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:07.287905 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:07.287912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:07.287930 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:22:07.288221 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.288233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.290028 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.290036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.291290 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.291305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.292393 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.292397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.292551 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.292622 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.293180 (+ 558us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.293211 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:22:07.293302 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 02:22:07.293438 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:22:07.293790 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.286563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.9.231.65:35185)
0620 02:22:07.286782 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.286790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.287238 (+ 448us) server_negotiation.cc:407] Connection header received
0620 02:22:07.287592 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.287597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.287698 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.287800 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:07.288593 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.289815 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.291442 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.292180 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.292335 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.292873 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.293021 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.293127 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.293348 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.293467 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:22:07.298631 18085 raft_consensus.cc:1261] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Refusing update from remote peer 142a6374f0d0478d8ca4c73c0a8ff2c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:07.298679 18222 raft_consensus.cc:1261] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Refusing update from remote peer 142a6374f0d0478d8ca4c73c0a8ff2c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:07.299365 18309 consensus_queue.cc:1035] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:07.299778 18342 consensus_queue.cc:1035] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:07.309877 18305 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: TxnStatusTablet state changed. Reason: New leader 142a6374f0d0478d8ca4c73c0a8ff2c4. Latest consensus state: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:07.310902 18305 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:22:07.319679 18305 consensus_queue.cc:1035] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:07.323470 18347 mvcc.cc:205] Tried to move back new op lower bound from 7298753033407840256 to 7298753033069457408. Current Snapshot: MvccSnapshot[applied={T|T < 7298753033407840256}]
I20260620 02:22:07.330173 18342 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:07.330816 18342 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:22:07.335645 18305 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:07.335987 18305 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 0d9c7aefb7d044b4af2c26ef7c09f096: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:22:07.346170 18306 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: TxnStatusTablet state changed. Reason: New leader 142a6374f0d0478d8ca4c73c0a8ff2c4. Latest consensus state: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:07.346472 18306 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:22:07.354542 18309 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:07.355022 18309 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 142a6374f0d0478d8ca4c73c0a8ff2c4: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:22:07.378548 18359 consensus_queue.cc:1035] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:07.380415 18306 tablet_replica.cc:439] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } } }
I20260620 02:22:07.380707 18306 tablet_replica.cc:442] T 41aae64e47f4485881c45aff5bd219aa P 8a1596d961cd4d3cbfe77307ec36b02a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:22:07.384865 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.346480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.9.231.66:43155)
0620 02:22:07.374917 (+ 28437us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.374927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.376467 (+ 1540us) server_negotiation.cc:407] Connection header received
0620 02:22:07.376577 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.376585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.376682 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.376863 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:07.377680 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.379175 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.381249 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.381949 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.382086 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.382804 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.383643 (+ 839us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.383794 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.384392 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.384496 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28250,"thread_start_us":29576,"threads_started":1}
I20260620 02:22:07.403640 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.397450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:42875 (local address 127.9.231.65:35185)
0620 02:22:07.397748 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.397761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.398204 (+ 443us) server_negotiation.cc:407] Connection header received
0620 02:22:07.398456 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.398460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.398533 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.398636 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:07.399234 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.400082 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.401559 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.402054 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.402201 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.402705 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.402890 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.402999 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.403167 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.403263 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:22:07.403692 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.397360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35185 (local address 127.9.231.66:42875)
0620 02:22:07.398080 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:07.398101 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:07.398295 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:07.398668 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:07.398673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:07.398687 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:22:07.398988 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.398997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.400243 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.400249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.401369 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.401383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.402233 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.402240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.402428 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.402515 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.402995 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.403008 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:22:07.403138 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:22:07.403302 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":354,"threads_started":1}
I20260620 02:22:07.452826 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.445573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:47029 (local address 127.9.231.66:43155)
0620 02:22:07.445788 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.445796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.446246 (+ 450us) server_negotiation.cc:407] Connection header received
0620 02:22:07.446449 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.446455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.446547 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.446647 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:07.447357 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.448548 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.450564 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.451286 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.451432 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.452162 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.452282 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.452357 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.452535 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.452611 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:22:07.453385 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.445125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:43155 (local address 127.9.231.65:47029)
0620 02:22:07.446125 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:07.446145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:07.446298 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:07.446662 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:07.446667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:07.446684 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:22:07.447189 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.447208 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.448694 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.448698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.450393 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.450407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.451595 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.451603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.451804 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.451891 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.452402 (+ 511us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.452415 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:22:07.452502 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 02:22:07.452674 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":673,"threads_started":1}
I20260620 02:22:07.492803 17783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:07.493110 17783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:07.497035 17783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Sending DeleteTablet for 3 replicas of tablet 506bea39fd2e4f0889df140eb082d9d2
I20260620 02:22:07.498116 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet 506bea39fd2e4f0889df140eb082d9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:07 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:41696
I20260620 02:22:07.498118 17921 tablet_service.cc:1514] Processing DeleteTablet for tablet 506bea39fd2e4f0889df140eb082d9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:07 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:52222
I20260620 02:22:07.498294 18202 tablet_service.cc:1514] Processing DeleteTablet for tablet 506bea39fd2e4f0889df140eb082d9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:07 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:58186
I20260620 02:22:07.518514 18372 tablet_replica.cc:330] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: stopping tablet replica
I20260620 02:22:07.518867 18372 raft_consensus.cc:2229] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:22:07.520282 18372 raft_consensus.cc:2258] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:07.523327 18374 tablet_replica.cc:330] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: stopping tablet replica
I20260620 02:22:07.523834 18374 raft_consensus.cc:2229] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:07.524220 18374 raft_consensus.cc:2258] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:07.525874 18374 ts_tablet_manager.cc:1905] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:07.526340 18372 ts_tablet_manager.cc:1905] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:07.531138 18373 tablet_replica.cc:330] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: stopping tablet replica
I20260620 02:22:07.531426 18373 raft_consensus.cc:2229] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:07.531684 18373 raft_consensus.cc:2258] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:07.532959 18373 ts_tablet_manager.cc:1905] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:07.534415 18374 ts_tablet_manager.cc:1918] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:22:07.534646 18374 log.cc:1196] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/506bea39fd2e4f0889df140eb082d9d2
I20260620 02:22:07.535285 18374 ts_tablet_manager.cc:1939] T 506bea39fd2e4f0889df140eb082d9d2 P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting consensus metadata
I20260620 02:22:07.536675 17767 catalog_manager.cc:4841] TS 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219): tablet 506bea39fd2e4f0889df140eb082d9d2 (table test-table [id=4b380676c77d469392ed13f5dff2f76e]) successfully deleted
I20260620 02:22:07.541582 18373 ts_tablet_manager.cc:1918] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:22:07.541780 18373 log.cc:1196] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/506bea39fd2e4f0889df140eb082d9d2
I20260620 02:22:07.542321 18373 ts_tablet_manager.cc:1939] T 506bea39fd2e4f0889df140eb082d9d2 P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting consensus metadata
I20260620 02:22:07.543725 17769 catalog_manager.cc:4841] TS 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185): tablet 506bea39fd2e4f0889df140eb082d9d2 (table test-table [id=4b380676c77d469392ed13f5dff2f76e]) successfully deleted
I20260620 02:22:07.545943 18372 ts_tablet_manager.cc:1918] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:22:07.546207 18372 log.cc:1196] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/506bea39fd2e4f0889df140eb082d9d2
I20260620 02:22:07.546718 18372 ts_tablet_manager.cc:1939] T 506bea39fd2e4f0889df140eb082d9d2 P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting consensus metadata
I20260620 02:22:07.548089 17769 catalog_manager.cc:4841] TS 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155): tablet 506bea39fd2e4f0889df140eb082d9d2 (table test-table [id=4b380676c77d469392ed13f5dff2f76e]) successfully deleted
Jun 20 02:22:07 dist-test-slave-9vg1 krb5kdc[17738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:22:07 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:07 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:22:07.632460 18382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:07.605104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.9.231.126:39019)
0620 02:22:07.617557 (+ 12453us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.617565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.618007 (+ 442us) server_negotiation.cc:407] Connection header received
0620 02:22:07.618248 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.618254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.618357 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.618497 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:07.620477 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.621724 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.623751 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.624435 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.624598 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.632127 (+ 7529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43684: BlockingRecv error: recv got EOF from 127.0.0.1:43684 (error 108)
Metrics: {"server-negotiator.queue_time_us":12271,"thread_start_us":12032,"threads_started":1}
Jun 20 02:22:07 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM
I20260620 02:22:07.656062 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.635944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.9.231.126:39019)
0620 02:22:07.636147 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.636154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.636666 (+ 512us) server_negotiation.cc:407] Connection header received
0620 02:22:07.636983 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.636991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.637083 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.637206 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:07.638787 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.639999 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.642062 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.642729 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.642890 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.648311 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:07.648320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:07.648325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:07.648396 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:07.652530 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:07.653627 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:07.653634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:07.653638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:07.653824 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:07.654321 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:07.654327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:07.654331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:07.654732 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:07.655008 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.655423 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.655691 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:22:07.660894 17783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:22:07.661717 17783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:22:07.670591 18202 tablet_service.cc:1467] Processing CreateTablet for tablet 387a58dc42cc48108165c5ed1bdd111e (DEFAULT_TABLE table=test-table [id=711cc4ca839a4f0db8481a66cbaf77d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:07.671150 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387a58dc42cc48108165c5ed1bdd111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:07.672361 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 387a58dc42cc48108165c5ed1bdd111e (DEFAULT_TABLE table=test-table [id=711cc4ca839a4f0db8481a66cbaf77d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:07.672808 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387a58dc42cc48108165c5ed1bdd111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:07.677589 17921 tablet_service.cc:1467] Processing CreateTablet for tablet 387a58dc42cc48108165c5ed1bdd111e (DEFAULT_TABLE table=test-table [id=711cc4ca839a4f0db8481a66cbaf77d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:07.678122 17921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387a58dc42cc48108165c5ed1bdd111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:07.688243 18389 tablet_bootstrap.cc:492] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: Bootstrap starting.
I20260620 02:22:07.691864 18390 tablet_bootstrap.cc:492] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: Bootstrap starting.
I20260620 02:22:07.693786 18390 tablet_bootstrap.cc:654] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:07.694403 18388 tablet_bootstrap.cc:492] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: Bootstrap starting.
I20260620 02:22:07.697108 18388 tablet_bootstrap.cc:654] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:07.702745 18390 tablet_bootstrap.cc:492] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: No bootstrap required, opened a new log
I20260620 02:22:07.703042 18390 ts_tablet_manager.cc:1397] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260620 02:22:07.703370 18389 tablet_bootstrap.cc:654] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:07.704247 18390 raft_consensus.cc:348] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.704672 18390 raft_consensus.cc:374] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:07.704814 18390 raft_consensus.cc:729] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1596d961cd4d3cbfe77307ec36b02a, State: Initialized, Role: FOLLOWER
I20260620 02:22:07.705168 18390 consensus_queue.cc:260] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.708637 18390 ts_tablet_manager.cc:1428] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 02:22:07.713721 18388 tablet_bootstrap.cc:492] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: No bootstrap required, opened a new log
I20260620 02:22:07.713920 18388 ts_tablet_manager.cc:1397] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20260620 02:22:07.714850 18388 raft_consensus.cc:348] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.715188 18388 raft_consensus.cc:374] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:07.715309 18388 raft_consensus.cc:729] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9c7aefb7d044b4af2c26ef7c09f096, State: Initialized, Role: FOLLOWER
I20260620 02:22:07.715608 18388 consensus_queue.cc:260] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.716318 18388 ts_tablet_manager.cc:1428] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:22:07.717909 18389 tablet_bootstrap.cc:492] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: No bootstrap required, opened a new log
I20260620 02:22:07.718082 18389 ts_tablet_manager.cc:1397] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.003s
I20260620 02:22:07.718976 18389 raft_consensus.cc:348] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.719281 18389 raft_consensus.cc:374] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:07.719388 18389 raft_consensus.cc:729] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142a6374f0d0478d8ca4c73c0a8ff2c4, State: Initialized, Role: FOLLOWER
I20260620 02:22:07.719693 18389 consensus_queue.cc:260] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.722759 18389 ts_tablet_manager.cc:1428] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260620 02:22:07.890336 18306 raft_consensus.cc:482] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:22:07.890580 18306 raft_consensus.cc:504] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.892349 18306 leader_election.cc:288] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185), 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155)
I20260620 02:22:07.901273 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.893831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:51027 (local address 127.9.231.66:43155)
0620 02:22:07.894020 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.894026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.894675 (+ 649us) server_negotiation.cc:407] Connection header received
0620 02:22:07.894897 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.894904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.895000 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.895152 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:07.895691 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.896731 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.899187 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.900071 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.900307 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.900389 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.900764 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.900943 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:22:07.901330 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.893628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:43155 (local address 127.9.231.67:51027)
0620 02:22:07.894532 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:07.894554 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:07.894708 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:07.895167 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:07.895171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:07.895195 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:07.895517 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.895526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.896899 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.896904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.898960 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.898971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.900318 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.900325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.900545 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.900583 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:22:07.900699 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 02:22:07.900865 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":3,"thread_start_us":551,"threads_started":1}
I20260620 02:22:07.901405 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.892758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:35809 (local address 127.9.231.65:35185)
0620 02:22:07.893018 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.893028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.893565 (+ 537us) server_negotiation.cc:407] Connection header received
0620 02:22:07.893822 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.893832 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.893941 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.894023 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:07.894968 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.896116 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.898869 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.900159 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.900374 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.900468 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.900966 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.901138 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:22:07.902196 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.892695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:35185 (local address 127.9.231.67:35809)
0620 02:22:07.893423 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:07.893443 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:07.893621 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:07.894279 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:07.894283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:07.894309 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:07.894712 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.894733 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.896285 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.896291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.898617 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:07.898645 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.900564 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.900570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:07.900756 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.900785 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:22:07.901602 (+ 817us) client_negotiation.cc:240] Negotiation successful
0620 02:22:07.901759 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":33,"thread_start_us":481,"threads_started":1}
I20260620 02:22:07.902537 18084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a58dc42cc48108165c5ed1bdd111e" candidate_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" is_pre_election: true
I20260620 02:22:07.902886 18084 raft_consensus.cc:2454] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a1596d961cd4d3cbfe77307ec36b02a in term 0.
I20260620 02:22:07.902978 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a58dc42cc48108165c5ed1bdd111e" candidate_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" is_pre_election: true
I20260620 02:22:07.903298 17941 raft_consensus.cc:2454] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a1596d961cd4d3cbfe77307ec36b02a in term 0.
I20260620 02:22:07.903538 18154 leader_election.cc:302] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9c7aefb7d044b4af2c26ef7c09f096, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:07.904033 18306 raft_consensus.cc:2781] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:22:07.904165 18306 raft_consensus.cc:482] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:22:07.904258 18306 raft_consensus.cc:3037] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:07.906699 18306 raft_consensus.cc:504] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.907452 18306 leader_election.cc:288] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [CANDIDATE]: Term 1 election: Requested vote from peers 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185), 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155)
I20260620 02:22:07.908107 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a58dc42cc48108165c5ed1bdd111e" candidate_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4"
I20260620 02:22:07.908244 18084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387a58dc42cc48108165c5ed1bdd111e" candidate_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096"
I20260620 02:22:07.908382 17941 raft_consensus.cc:3037] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:07.908514 18084 raft_consensus.cc:3037] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:07.910594 17941 raft_consensus.cc:2454] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a1596d961cd4d3cbfe77307ec36b02a in term 1.
I20260620 02:22:07.911324 18154 leader_election.cc:302] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142a6374f0d0478d8ca4c73c0a8ff2c4, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:07.911679 18306 raft_consensus.cc:2781] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:07.912117 18084 raft_consensus.cc:2454] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a1596d961cd4d3cbfe77307ec36b02a in term 1.
I20260620 02:22:07.912253 18306 raft_consensus.cc:686] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 LEADER]: Becoming Leader. State: Replica: 8a1596d961cd4d3cbfe77307ec36b02a, State: Running, Role: LEADER
I20260620 02:22:07.912722 18306 consensus_queue.cc:237] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:07.917030 17783 catalog_manager.cc:5495] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67). New cstate: current_term: 1 leader_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } health_report { overall_health: HEALTHY } } }
I20260620 02:22:07.934376 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:07.927719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.9.231.67:46219)
0620 02:22:07.928360 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:07.928364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:07.928383 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:22:07.928474 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:07.928478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:07.928567 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:07.928748 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:07.929190 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.929933 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.931491 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:07.931934 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:07.932595 (+ 661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:07.932737 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:07.933510 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:07.933682 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:07.933834 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:22:07.933941 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":371,"threads_started":1}
I20260620 02:22:07.938639 18084 raft_consensus.cc:1261] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Refusing update from remote peer 8a1596d961cd4d3cbfe77307ec36b02a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:07.938747 17941 raft_consensus.cc:1261] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Refusing update from remote peer 8a1596d961cd4d3cbfe77307ec36b02a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:07.939533 18306 consensus_queue.cc:1035] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:07.939949 18396 consensus_queue.cc:1035] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [LEADER]: Connected to new peer: Peer: permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:07.946010 18349 mvcc.cc:205] Tried to move back new op lower bound from 7298753036028579840 to 7298753035934896128. Current Snapshot: MvccSnapshot[applied={T|T < 7298753036028579840 or (T in {7298753036028579840})}]
I20260620 02:22:07.969889 17783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:07.970194 17783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:07.973620 17783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Sending DeleteTablet for 3 replicas of tablet 387a58dc42cc48108165c5ed1bdd111e
I20260620 02:22:07.974866 18202 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a58dc42cc48108165c5ed1bdd111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:07 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:58186
I20260620 02:22:07.974958 17921 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a58dc42cc48108165c5ed1bdd111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:07 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:52222
I20260620 02:22:07.975291 18373 tablet_replica.cc:330] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: stopping tablet replica
I20260620 02:22:07.975315 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet 387a58dc42cc48108165c5ed1bdd111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:07 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:41696
I20260620 02:22:07.975514 18373 raft_consensus.cc:2229] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:07.975628 18374 tablet_replica.cc:330] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: stopping tablet replica
I20260620 02:22:07.975747 18373 raft_consensus.cc:2258] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:07.975787 18374 raft_consensus.cc:2229] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:22:07.976356 18374 raft_consensus.cc:2258] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:07.976684 18372 tablet_replica.cc:330] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: stopping tablet replica
I20260620 02:22:07.976920 18372 raft_consensus.cc:2229] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:07.977088 18373 ts_tablet_manager.cc:1905] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:07.977193 18372 raft_consensus.cc:2258] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:07.979687 17783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:22:07.980928 18374 ts_tablet_manager.cc:1905] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:07.983309 18372 ts_tablet_manager.cc:1905] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 02:22:07.984023 17783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:22:07.989717 18374 ts_tablet_manager.cc:1918] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:07.998461 18061 tablet_service.cc:1467] Processing CreateTablet for tablet ec4272e34a9d442c9930c0aac5c5616c (DEFAULT_TABLE table=test-table [id=7631266f4e9d4367954784e2ecbc9a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:07.998970 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4272e34a9d442c9930c0aac5c5616c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:08.001097 18373 ts_tablet_manager.cc:1918] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:08.001243 18373 log.cc:1196] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/387a58dc42cc48108165c5ed1bdd111e
I20260620 02:22:08.001679 18373 ts_tablet_manager.cc:1939] T 387a58dc42cc48108165c5ed1bdd111e P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting consensus metadata
I20260620 02:22:08.003796 17921 tablet_service.cc:1467] Processing CreateTablet for tablet ec4272e34a9d442c9930c0aac5c5616c (DEFAULT_TABLE table=test-table [id=7631266f4e9d4367954784e2ecbc9a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:08.004292 17921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4272e34a9d442c9930c0aac5c5616c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:08.012423 18372 ts_tablet_manager.cc:1918] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:08.012639 18372 log.cc:1196] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/387a58dc42cc48108165c5ed1bdd111e
I20260620 02:22:08.013084 18372 ts_tablet_manager.cc:1939] T 387a58dc42cc48108165c5ed1bdd111e P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting consensus metadata
I20260620 02:22:08.014351 17769 catalog_manager.cc:4841] TS 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155): tablet 387a58dc42cc48108165c5ed1bdd111e (table test-table [id=711cc4ca839a4f0db8481a66cbaf77d9]) successfully deleted
I20260620 02:22:08.015021 17769 catalog_manager.cc:4841] TS 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185): tablet 387a58dc42cc48108165c5ed1bdd111e (table test-table [id=711cc4ca839a4f0db8481a66cbaf77d9]) successfully deleted
I20260620 02:22:08.020509 18389 tablet_bootstrap.cc:492] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: Bootstrap starting.
I20260620 02:22:08.022431 18389 tablet_bootstrap.cc:654] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:08.025864 18388 tablet_bootstrap.cc:492] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: Bootstrap starting.
I20260620 02:22:08.028223 18388 tablet_bootstrap.cc:654] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:07.998267 18202 tablet_service.cc:1467] Processing CreateTablet for tablet ec4272e34a9d442c9930c0aac5c5616c (DEFAULT_TABLE table=test-table [id=7631266f4e9d4367954784e2ecbc9a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:08.034689 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4272e34a9d442c9930c0aac5c5616c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:08.036216 18374 log.cc:1196] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/387a58dc42cc48108165c5ed1bdd111e
I20260620 02:22:08.036878 18374 ts_tablet_manager.cc:1939] T 387a58dc42cc48108165c5ed1bdd111e P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting consensus metadata
I20260620 02:22:08.038352 17767 catalog_manager.cc:4841] TS 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219): tablet 387a58dc42cc48108165c5ed1bdd111e (table test-table [id=711cc4ca839a4f0db8481a66cbaf77d9]) successfully deleted
I20260620 02:22:08.041986 18389 tablet_bootstrap.cc:492] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: No bootstrap required, opened a new log
I20260620 02:22:08.042281 18389 ts_tablet_manager.cc:1397] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.008s
I20260620 02:22:08.043215 18390 tablet_bootstrap.cc:492] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: Bootstrap starting.
I20260620 02:22:08.043259 18389 raft_consensus.cc:348] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.043679 18389 raft_consensus.cc:374] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:08.043808 18389 raft_consensus.cc:729] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142a6374f0d0478d8ca4c73c0a8ff2c4, State: Initialized, Role: FOLLOWER
I20260620 02:22:08.044171 18389 consensus_queue.cc:260] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.044878 18389 ts_tablet_manager.cc:1428] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:22:08.045315 18390 tablet_bootstrap.cc:654] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:08.046602 18388 tablet_bootstrap.cc:492] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: No bootstrap required, opened a new log
I20260620 02:22:08.046797 18388 ts_tablet_manager.cc:1397] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.001s
I20260620 02:22:08.047664 18388 raft_consensus.cc:348] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.047940 18388 raft_consensus.cc:374] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:08.048036 18388 raft_consensus.cc:729] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9c7aefb7d044b4af2c26ef7c09f096, State: Initialized, Role: FOLLOWER
I20260620 02:22:08.048321 18388 consensus_queue.cc:260] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.048888 18388 ts_tablet_manager.cc:1428] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:22:08.063268 18390 tablet_bootstrap.cc:492] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: No bootstrap required, opened a new log
I20260620 02:22:08.063501 18390 ts_tablet_manager.cc:1397] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20260620 02:22:08.064639 18390 raft_consensus.cc:348] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.064918 18390 raft_consensus.cc:374] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:08.064999 18390 raft_consensus.cc:729] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1596d961cd4d3cbfe77307ec36b02a, State: Initialized, Role: FOLLOWER
I20260620 02:22:08.065258 18390 consensus_queue.cc:260] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.067243 18390 ts_tablet_manager.cc:1428] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:22:08.391290 18361 raft_consensus.cc:482] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:22:08.391598 18361 raft_consensus.cc:504] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.392387 18361 leader_election.cc:288] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:08.393029 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4272e34a9d442c9930c0aac5c5616c" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" is_pre_election: true
I20260620 02:22:08.393085 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4272e34a9d442c9930c0aac5c5616c" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" is_pre_election: true
I20260620 02:22:08.393342 17941 raft_consensus.cc:2454] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 0.
I20260620 02:22:08.393396 18222 raft_consensus.cc:2454] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 0.
I20260620 02:22:08.393890 18014 leader_election.cc:302] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9c7aefb7d044b4af2c26ef7c09f096, 142a6374f0d0478d8ca4c73c0a8ff2c4; no voters:
I20260620 02:22:08.394253 18361 raft_consensus.cc:2781] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:22:08.394384 18361 raft_consensus.cc:482] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:22:08.394495 18361 raft_consensus.cc:3037] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:08.396816 18361 raft_consensus.cc:504] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.397495 18361 leader_election.cc:288] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 election: Requested vote from peers 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185), 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219)
I20260620 02:22:08.398130 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4272e34a9d442c9930c0aac5c5616c" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1596d961cd4d3cbfe77307ec36b02a"
I20260620 02:22:08.398166 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4272e34a9d442c9930c0aac5c5616c" candidate_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4"
I20260620 02:22:08.398365 18222 raft_consensus.cc:3037] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:08.398427 17941 raft_consensus.cc:3037] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:08.400851 18222 raft_consensus.cc:2454] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 1.
I20260620 02:22:08.401463 18012 leader_election.cc:302] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9c7aefb7d044b4af2c26ef7c09f096, 8a1596d961cd4d3cbfe77307ec36b02a; no voters:
I20260620 02:22:08.401769 17941 raft_consensus.cc:2454] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9c7aefb7d044b4af2c26ef7c09f096 in term 1.
I20260620 02:22:08.402189 18361 raft_consensus.cc:2781] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:08.402808 18361 raft_consensus.cc:686] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9c7aefb7d044b4af2c26ef7c09f096, State: Running, Role: LEADER
I20260620 02:22:08.403435 18361 consensus_queue.cc:237] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } }
I20260620 02:22:08.406724 17783 catalog_manager.cc:5495] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9c7aefb7d044b4af2c26ef7c09f096" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 43155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 } health_report { overall_health: UNKNOWN } } }
I20260620 02:22:08.506044 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:08.500017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.9.231.66:43155)
0620 02:22:08.500784 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:08.500788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:08.500810 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:22:08.500876 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:08.500879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:08.500945 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:08.501030 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:08.501535 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.502391 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.504017 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.504517 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.504686 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:08.505166 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:08.505284 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:08.505399 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:08.505559 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 02:22:08.505691 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":478,"threads_started":1}
I20260620 02:22:08.517050 18222 raft_consensus.cc:1261] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Refusing update from remote peer 0d9c7aefb7d044b4af2c26ef7c09f096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:08.517148 17941 raft_consensus.cc:1261] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Refusing update from remote peer 0d9c7aefb7d044b4af2c26ef7c09f096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:08.517793 18407 consensus_queue.cc:1035] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142a6374f0d0478d8ca4c73c0a8ff2c4" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:08.518211 18361 consensus_queue.cc:1035] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1596d961cd4d3cbfe77307ec36b02a" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:08.600404 17780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:08.600653 17780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:08.603916 17780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b607451c74364485a977b236ec946af4: Sending DeleteTablet for 3 replicas of tablet ec4272e34a9d442c9930c0aac5c5616c
I20260620 02:22:08.604888 17921 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4272e34a9d442c9930c0aac5c5616c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:08 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:52222
I20260620 02:22:08.605010 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4272e34a9d442c9930c0aac5c5616c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:08 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:41696
I20260620 02:22:08.605068 18202 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4272e34a9d442c9930c0aac5c5616c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:08 UTC) from {username='oryx', principal='oryx/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:58186
I20260620 02:22:08.605811 18432 tablet_replica.cc:330] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: stopping tablet replica
I20260620 02:22:08.606084 18431 tablet_replica.cc:330] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: stopping tablet replica
I20260620 02:22:08.606081 18432 raft_consensus.cc:2229] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:22:08.606278 18431 raft_consensus.cc:2229] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:08.606523 18431 raft_consensus.cc:2258] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:08.606712 18432 raft_consensus.cc:2258] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:08.606877 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 17854
I20260620 02:22:08.608222 18432 ts_tablet_manager.cc:1905] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:08.608907 18431 ts_tablet_manager.cc:1905] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:08.615976 18432 ts_tablet_manager.cc:1918] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:22:08.615983 18431 ts_tablet_manager.cc:1918] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:22:08.616138 18431 log.cc:1196] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/ec4272e34a9d442c9930c0aac5c5616c
I20260620 02:22:08.616307 18432 log.cc:1196] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/ec4272e34a9d442c9930c0aac5c5616c
I20260620 02:22:08.616626 18431 ts_tablet_manager.cc:1939] T ec4272e34a9d442c9930c0aac5c5616c P 142a6374f0d0478d8ca4c73c0a8ff2c4: Deleting consensus metadata
I20260620 02:22:08.616959 18432 ts_tablet_manager.cc:1939] T ec4272e34a9d442c9930c0aac5c5616c P 0d9c7aefb7d044b4af2c26ef7c09f096: Deleting consensus metadata
I20260620 02:22:08.618038 18433 tablet_replica.cc:330] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: stopping tablet replica
I20260620 02:22:08.618144 17769 catalog_manager.cc:4841] TS 0d9c7aefb7d044b4af2c26ef7c09f096 (127.9.231.66:43155): tablet ec4272e34a9d442c9930c0aac5c5616c (table test-table [id=7631266f4e9d4367954784e2ecbc9a68]) successfully deleted
I20260620 02:22:08.618294 18433 raft_consensus.cc:2229] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:08.618551 18433 raft_consensus.cc:2258] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:08.618510 17769 catalog_manager.cc:4841] TS 142a6374f0d0478d8ca4c73c0a8ff2c4 (127.9.231.65:35185): tablet ec4272e34a9d442c9930c0aac5c5616c (table test-table [id=7631266f4e9d4367954784e2ecbc9a68]) successfully deleted
I20260620 02:22:08.619683 18433 ts_tablet_manager.cc:1905] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:08.624490 18433 ts_tablet_manager.cc:1918] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:22:08.624672 18433 log.cc:1196] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/ec4272e34a9d442c9930c0aac5c5616c
I20260620 02:22:08.625197 18433 ts_tablet_manager.cc:1939] T ec4272e34a9d442c9930c0aac5c5616c P 8a1596d961cd4d3cbfe77307ec36b02a: Deleting consensus metadata
I20260620 02:22:08.626358 17767 catalog_manager.cc:4841] TS 8a1596d961cd4d3cbfe77307ec36b02a (127.9.231.67:46219): tablet ec4272e34a9d442c9930c0aac5c5616c (table test-table [id=7631266f4e9d4367954784e2ecbc9a68]) successfully deleted
Jun 20 02:22:08 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.65@KRBTEST.COM
I20260620 02:22:08.635977 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:08.608978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.9.231.65:35185)
0620 02:22:08.615534 (+ 6556us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:08.615544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:08.619340 (+ 3796us) server_negotiation.cc:407] Connection header received
0620 02:22:08.619566 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:08.619571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:08.619649 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:08.619771 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:08.621019 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.621800 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.623041 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.623562 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.623709 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:08.628913 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:08.628932 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:08.628949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:08.628998 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:08.632422 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:08.633325 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:08.633333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:08.633336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:08.633487 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:08.633972 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:08.633980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:08.633984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:08.634379 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:08.634725 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:08.635250 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 02:22:08.635545 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6351,"thread_start_us":6191,"threads_started":1}
I20260620 02:22:08.636746 17986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:08.788247 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 17854
I20260620 02:22:08.816727 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 17995
Jun 20 02:22:08 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.66@KRBTEST.COM
I20260620 02:22:08.833151 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:08.817541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.9.231.66:43155)
0620 02:22:08.817724 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:08.817729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:08.817751 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:22:08.817929 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:08.817933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:08.817999 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:08.818096 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:08.819445 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.820375 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.821712 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.822151 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.822261 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:08.826883 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:08.826901 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:08.826918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:08.826962 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:08.829931 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:08.830770 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:08.830779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:08.830782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:08.830935 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:08.831384 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:08.831391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:08.831396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:08.831875 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:08.832213 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:08.832571 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 02:22:08.832807 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:22:08.833943 18126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:08.966491 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 17995
I20260620 02:22:08.990180 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 18135
Jun 20 02:22:08 dist-test-slave-9vg1 krb5kdc[17738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.231.67@KRBTEST.COM
I20260620 02:22:09.007222 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:08.991043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.9.231.67:46219)
0620 02:22:08.991616 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:08.991620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:08.991635 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:22:08.991688 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:08.991692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:08.991750 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:08.991887 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:08.993178 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.993936 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.995278 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:08.995693 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:08.995841 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:09.000714 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:09.000732 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:09.000751 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:09.000799 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:09.004162 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:09.005033 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:09.005042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:09.005044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:09.005195 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:09.005595 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:09.005599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:09.005601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:09.006022 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:09.006269 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:09.006649 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 02:22:09.006854 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":261,"threads_started":1}
I20260620 02:22:09.008034 18267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:09.135740 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 18135
I20260620 02:22:09.159013 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 17751
I20260620 02:22:09.160446 17823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:09.280828 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 17751
2026-06-20T02:22:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7866 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:22:09 dist-test-slave-9vg1 krb5kdc[18443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:22:09 dist-test-slave-9vg1 krb5kdc[18443](info): set up 2 sockets
Jun 20 02:22:09 dist-test-slave-9vg1 krb5kdc[18443](info): commencing operation
krb5kdc: starting...
W20260620 02:22:10.401376 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:22:10 dist-test-slave-9vg1 krb5kdc[18443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:22:10.656881 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:41129
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:11.010344 18456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:11.010758 18456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:11.010967 18456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:11.018890 18456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:22:11.019145 18456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:11.019207 18456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:11.019268 18456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:22:11.019316 18456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:22:11.030237 18456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:41129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:11.031718 18456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:11.033128 18456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:11.046002 18464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:11.046161 18461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:11.046162 18462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:11.046823 18456 server_base.cc:1034] running on GCE node
I20260620 02:22:11.047844 18456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:11.049448 18456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:11.050683 18456 hybrid_clock.cc:648] HybridClock initialized: now 1781922131050616 us; error 59 us; skew 500 ppm
I20260620 02:22:11.051211 18456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T02:22:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1834 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:22:11 dist-test-slave-9vg1 krb5kdc[18470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:22:11 dist-test-slave-9vg1 krb5kdc[18470](info): set up 2 sockets
Jun 20 02:22:11 dist-test-slave-9vg1 krb5kdc[18470](info): commencing operation
krb5kdc: starting...
W20260620 02:22:12.241185 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:22:12 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:22:12.487978 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:43409
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:43409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:12.841410 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:12.841773 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:12.841943 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:12.849969 18483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:22:12.850112 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:12.850162 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:12.850276 18483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:22:12.850342 18483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:22:12.863283 18483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:43409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:12.865090 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:12.866431 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:12.879487 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:12.879422 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:12.879442 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:12.880334 18483 server_base.cc:1034] running on GCE node
I20260620 02:22:12.881337 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:12.883253 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:12.884523 18483 hybrid_clock.cc:648] HybridClock initialized: now 1781922132884432 us; error 62 us; skew 500 ppm
I20260620 02:22:12.884999 18483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:12 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:12.895287 18483 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:22:12.897181 18483 webserver.cc:460] Webserver started at http://127.9.231.126:33621/ using document root <none> and password file <none>
I20260620 02:22:12.898012 18483 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:12.898183 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:12.898540 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:12.902406 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "f878be27a4154fe2a980355130f56179"
format_stamp: "Formatted at 2026-06-20 02:22:12 on dist-test-slave-9vg1"
server_key: "2d67ddeea89194e8808ab6793a3e8fe6"
server_key_iv: "253896c6d1e8f131e758e2eedcd1950c"
server_key_version: "encryptionkey@0"
I20260620 02:22:12.903368 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "f878be27a4154fe2a980355130f56179"
format_stamp: "Formatted at 2026-06-20 02:22:12 on dist-test-slave-9vg1"
server_key: "2d67ddeea89194e8808ab6793a3e8fe6"
server_key_iv: "253896c6d1e8f131e758e2eedcd1950c"
server_key_version: "encryptionkey@0"
I20260620 02:22:12.909451 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:22:12.913648 18498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:12.915445 18483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:22:12.915668 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "f878be27a4154fe2a980355130f56179"
format_stamp: "Formatted at 2026-06-20 02:22:12 on dist-test-slave-9vg1"
server_key: "2d67ddeea89194e8808ab6793a3e8fe6"
server_key_iv: "253896c6d1e8f131e758e2eedcd1950c"
server_key_version: "encryptionkey@0"
I20260620 02:22:12.915908 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:12.944762 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:12.949733 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:12.950089 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:12.971843 18483 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:43409
I20260620 02:22:12.971859 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:43409 every 8 connection(s)
I20260620 02:22:12.973542 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:22:12.977734 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:12.983428 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 18483
I20260620 02:22:12.983691 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:22:12.984397 10141 external_mini_cluster.cc:1428] Setting key 074df7c482bbbec2aaa09c531014a5cc
I20260620 02:22:12.989751 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Bootstrap starting.
I20260620 02:22:12.993732 18551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:12.995128 18551 log.cc:824] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:12.998705 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: No bootstrap required, opened a new log
Jun 20 02:22:12 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:13.007161 18551 raft_consensus.cc:348] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:13.007570 18551 raft_consensus.cc:374] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:13.007691 18551 raft_consensus.cc:729] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f878be27a4154fe2a980355130f56179, State: Initialized, Role: FOLLOWER
I20260620 02:22:13.008397 18551 consensus_queue.cc:260] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:13.008673 18551 raft_consensus.cc:388] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:22:13.008855 18551 raft_consensus.cc:482] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:22:13.009052 18551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:13.010257 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:12.986780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.9.231.126:43409)
0620 02:22:12.987816 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:12.987831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:12.987877 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:22:12.989663 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:12.989688 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:12.990178 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:12.990552 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:12.992595 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:12.994078 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:12.995973 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:12.996983 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:12.997135 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:13.001833 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:13.001842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:13.001864 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:13.001910 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:13.005429 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:13.006317 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:13.006324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:13.006332 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:13.006514 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:13.006926 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:13.006930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:13.006932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:13.007605 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:13.007953 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:13.008415 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 02:22:13.008917 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":515,"threads_started":1}
I20260620 02:22:13.011739 18551 raft_consensus.cc:504] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:13.012305 18551 leader_election.cc:302] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f878be27a4154fe2a980355130f56179; no voters:
I20260620 02:22:13.013020 18551 leader_election.cc:288] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:22:13.013131 18556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:13.014654 18556 raft_consensus.cc:686] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 LEADER]: Becoming Leader. State: Replica: f878be27a4154fe2a980355130f56179, State: Running, Role: LEADER
I20260620 02:22:13.015316 18556 consensus_queue.cc:237] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:13.016022 18551 sys_catalog.cc:564] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:22:13.019230 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f878be27a4154fe2a980355130f56179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } } }
I20260620 02:22:13.019263 18558 sys_catalog.cc:455] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f878be27a4154fe2a980355130f56179. Latest consensus state: current_term: 1 leader_uuid: "f878be27a4154fe2a980355130f56179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } } }
I20260620 02:22:13.019802 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:13.019809 18558 sys_catalog.cc:458] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:13.021483 18562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:22:13.029109 18562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:22:13.040479 18562 catalog_manager.cc:1347] Generated new cluster ID: 8cdc90b714ca4d76a4e04bb53a046ba4
I20260620 02:22:13.040616 18562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:22:13.071013 18562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:22:13.072518 18562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:22:13.086942 18562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Generated new TSK 0
I20260620 02:22:13.088184 18562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:22:13.160161 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:43409
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:13.528762 18577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:13.529078 18577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:13.529229 18577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:13.536840 18577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:13.536963 18577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:13.537137 18577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:22:13.547310 18577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:13.548695 18577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:13.550038 18577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:13.564035 18582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:13.564064 18583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:13.564687 18585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:13.565095 18577 server_base.cc:1034] running on GCE node
I20260620 02:22:13.565742 18577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:13.566764 18577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:13.568011 18577 hybrid_clock.cc:648] HybridClock initialized: now 1781922133567944 us; error 51 us; skew 500 ppm
I20260620 02:22:13.568435 18577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:13 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922133, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:13.572119 18577 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:22:13.573702 18577 webserver.cc:460] Webserver started at http://127.9.231.65:36927/ using document root <none> and password file <none>
I20260620 02:22:13.574436 18577 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:13.574564 18577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:13.574909 18577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:13.578058 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "4de3cee863d84a4697d5cd9c40845798"
format_stamp: "Formatted at 2026-06-20 02:22:13 on dist-test-slave-9vg1"
server_key: "7e32bf6efe00bdf60c7eefa47ea00d7a"
server_key_iv: "d2761d4515b7ecc0df8c5c8a46f4270c"
server_key_version: "encryptionkey@0"
I20260620 02:22:13.578981 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "4de3cee863d84a4697d5cd9c40845798"
format_stamp: "Formatted at 2026-06-20 02:22:13 on dist-test-slave-9vg1"
server_key: "7e32bf6efe00bdf60c7eefa47ea00d7a"
server_key_iv: "d2761d4515b7ecc0df8c5c8a46f4270c"
server_key_version: "encryptionkey@0"
I20260620 02:22:13.584674 18577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:22:13.588891 18593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:13.590580 18577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:22:13.590767 18577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "4de3cee863d84a4697d5cd9c40845798"
format_stamp: "Formatted at 2026-06-20 02:22:13 on dist-test-slave-9vg1"
server_key: "7e32bf6efe00bdf60c7eefa47ea00d7a"
server_key_iv: "d2761d4515b7ecc0df8c5c8a46f4270c"
server_key_version: "encryptionkey@0"
I20260620 02:22:13.591004 18577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:13.609768 18577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:13.614702 18577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:13.615038 18577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:13.616127 18577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:13.617758 18577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:13.617875 18577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:13.618022 18577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:13.618093 18577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:13.654575 18577 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:45209
I20260620 02:22:13.654621 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:45209 every 8 connection(s)
I20260620 02:22:13.655944 18577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:22:13.659400 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 18577
I20260620 02:22:13.659677 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:22:13.660463 10141 external_mini_cluster.cc:1428] Setting key 54189544d42a97dc2654c58e548a2750
Jun 20 02:22:13 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922133, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:13.677487 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:13.658577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:44777 (local address 127.9.231.126:43409)
0620 02:22:13.659039 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:13.659043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:13.660138 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 02:22:13.662006 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:13.662012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:13.662101 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:13.662216 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:13.663808 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:13.664552 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:13.666153 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:13.666546 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:13.666679 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:13.670710 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:13.670717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:13.670720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:13.670819 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:13.673771 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:13.674719 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:13.674731 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:13.674733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:13.674892 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:13.675463 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:13.675468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:13.675470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:13.675791 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:13.676070 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:13.676857 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 02:22:13.677105 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":251,"threads_started":1}
I20260620 02:22:13.678527 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:13.659051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:44777)
0620 02:22:13.659967 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:13.660019 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:13.661731 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:13.662462 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:13.662470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:13.662982 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:13.663634 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:13.663651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:13.664723 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:13.664727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:13.665984 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:13.665997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:13.666795 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:13.666802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:13.667098 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:13.667841 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:13.667869 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:13.670415 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:13.673970 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:13.673981 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:13.674003 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:13.674562 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:13.675030 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:13.675036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:13.675040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:13.675316 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:13.676031 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:13.676040 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:13.676444 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 02:22:13.676730 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 02:22:13.677308 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":343,"threads_started":1}
I20260620 02:22:13.680274 18706 heartbeater.cc:347] Connected to a master server at 127.9.231.126:43409
I20260620 02:22:13.680665 18706 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:13.681685 18706 heartbeater.cc:511] Master 127.9.231.126:43409 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:22:13.684324 18516 ts_manager.cc:194] Registered new tserver with Master: 4de3cee863d84a4697d5cd9c40845798 (127.9.231.65:45209)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:22:13.686046 18516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:44777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:22:13.715888 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:43409
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:14.067698 18712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:14.068025 18712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:14.068192 18712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:14.076005 18712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:14.076124 18712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:14.076298 18712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:22:14.086892 18712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:14.088444 18712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:14.089870 18712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:14.104375 18720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:14.104525 18717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:14.104405 18718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:14.105144 18712 server_base.cc:1034] running on GCE node
I20260620 02:22:14.105885 18712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:14.107098 18712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:14.108366 18712 hybrid_clock.cc:648] HybridClock initialized: now 1781922134108296 us; error 65 us; skew 500 ppm
I20260620 02:22:14.108722 18712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:14 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:14.112227 18712 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:22:14.113876 18712 webserver.cc:460] Webserver started at http://127.9.231.66:36055/ using document root <none> and password file <none>
I20260620 02:22:14.114621 18712 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:14.114730 18712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:14.115021 18712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:14.118428 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "1d17f7e35d084c9a987f9c05ac3a1001"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "4ed6bbc9d4ae7b2d455644e40d3a75d8"
server_key_iv: "07be752d7b8eb313567d9bc42fe08fca"
server_key_version: "encryptionkey@0"
I20260620 02:22:14.119447 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "1d17f7e35d084c9a987f9c05ac3a1001"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "4ed6bbc9d4ae7b2d455644e40d3a75d8"
server_key_iv: "07be752d7b8eb313567d9bc42fe08fca"
server_key_version: "encryptionkey@0"
I20260620 02:22:14.125423 18712 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:22:14.129568 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:14.131243 18712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:22:14.131420 18712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "1d17f7e35d084c9a987f9c05ac3a1001"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "4ed6bbc9d4ae7b2d455644e40d3a75d8"
server_key_iv: "07be752d7b8eb313567d9bc42fe08fca"
server_key_version: "encryptionkey@0"
I20260620 02:22:14.131633 18712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:14.152696 18712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:14.157565 18712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:14.157915 18712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:14.159045 18712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:14.160604 18712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:14.160682 18712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:14.160856 18712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:14.160925 18712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:14.196127 18712 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:35053
I20260620 02:22:14.196223 18840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:35053 every 8 connection(s)
I20260620 02:22:14.197482 18712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:22:14.201275 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 18712
I20260620 02:22:14.201498 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:22:14.202024 10141 external_mini_cluster.cc:1428] Setting key 64fc91e3fe8451076f7c6ece27105ff2
Jun 20 02:22:14 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:14.219533 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:14.200391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:34473 (local address 127.9.231.126:43409)
0620 02:22:14.200888 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:14.200893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:14.201965 (+ 1072us) server_negotiation.cc:407] Connection header received
0620 02:22:14.204206 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:14.204212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:14.204295 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:14.204429 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:14.206150 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.206876 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.208584 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.209039 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.209178 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:14.213191 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:14.213197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:14.213200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:14.213236 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:14.216073 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:14.216989 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:14.216995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:14.217003 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:14.217134 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:14.217625 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:14.217636 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:14.217639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:14.217938 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:14.218120 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:14.218845 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 02:22:14.219111 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
I20260620 02:22:14.220661 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:14.200830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:34473)
0620 02:22:14.201733 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:14.201776 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:14.203935 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:14.204736 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:14.204749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:14.205366 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:14.205976 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:14.205994 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.207051 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.207097 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:14.208419 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:14.208433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.209299 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.209307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:14.209560 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:14.210305 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:14.210336 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:14.212910 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:14.216336 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:14.216347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:14.216368 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:14.216843 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:14.217249 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:14.217255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:14.217258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:14.217504 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:14.218078 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:14.218087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:14.218473 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 02:22:14.218747 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 02:22:14.219368 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":324,"threads_started":1}
I20260620 02:22:14.222132 18841 heartbeater.cc:347] Connected to a master server at 127.9.231.126:43409
I20260620 02:22:14.222458 18841 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:14.223289 18841 heartbeater.cc:511] Master 127.9.231.126:43409 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
I20260620 02:22:14.224987 18516 ts_manager.cc:194] Registered new tserver with Master: 1d17f7e35d084c9a987f9c05ac3a1001 (127.9.231.66:35053)
I20260620 02:22:14.225936 18516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:34473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:22:14.255033 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:43409
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:14.603376 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:14.603693 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:14.603788 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:14.611398 18847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:14.611523 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:14.611639 18847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:22:14.622227 18847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:14.623622 18847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:14.624976 18847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:14.640295 18852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:14.640348 18853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:14.640254 18855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:14.641343 18847 server_base.cc:1034] running on GCE node
I20260620 02:22:14.642207 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:14.643610 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:14.644948 18847 hybrid_clock.cc:648] HybridClock initialized: now 1781922134644894 us; error 61 us; skew 500 ppm
I20260620 02:22:14.645365 18847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:14 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:14.649132 18847 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:22:14.650877 18847 webserver.cc:460] Webserver started at http://127.9.231.67:36407/ using document root <none> and password file <none>
I20260620 02:22:14.651669 18847 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:14.651784 18847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:14.652078 18847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:14.655493 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "8e5a75adafa046c380d77b553d8c0920"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "6ca2fb7dee6b86ee58778e4bea900cf5"
server_key_iv: "e7f2443b5d8a8169bf4b85c01f8a32c6"
server_key_version: "encryptionkey@0"
I20260620 02:22:14.656472 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "8e5a75adafa046c380d77b553d8c0920"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "6ca2fb7dee6b86ee58778e4bea900cf5"
server_key_iv: "e7f2443b5d8a8169bf4b85c01f8a32c6"
server_key_version: "encryptionkey@0"
I20260620 02:22:14.662436 18847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:22:14.666731 18862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:14.668670 18847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:22:14.668910 18847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "8e5a75adafa046c380d77b553d8c0920"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "6ca2fb7dee6b86ee58778e4bea900cf5"
server_key_iv: "e7f2443b5d8a8169bf4b85c01f8a32c6"
server_key_version: "encryptionkey@0"
I20260620 02:22:14.669150 18847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:14.689492 18706 heartbeater.cc:503] Master 127.9.231.126:43409 was elected leader, sending a full tablet report...
I20260620 02:22:14.691959 18847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:14.696966 18847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:14.697285 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:14.698390 18847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:14.700071 18847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:14.700186 18847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:14.700335 18847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:14.700407 18847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:14.736442 18847 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:46095
I20260620 02:22:14.736474 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:46095 every 8 connection(s)
I20260620 02:22:14.737864 18847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:22:14.739426 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 18847
I20260620 02:22:14.739636 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:22:14.740152 10141 external_mini_cluster.cc:1428] Setting key 4688d157c441acc4725da461c0ba26df
Jun 20 02:22:14 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922134, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:14.761395 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:14.741346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:42211 (local address 127.9.231.126:43409)
0620 02:22:14.741858 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:14.741866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:14.742875 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 02:22:14.745103 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:14.745109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:14.745212 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:14.745330 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:14.747142 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.747928 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.749651 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.750108 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.750259 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:14.754357 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:14.754363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:14.754366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:14.754440 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:14.757856 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:14.758743 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:14.758748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:14.758751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:14.758882 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:14.759461 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:14.759466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:14.759469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:14.759796 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:14.759947 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:14.760865 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 02:22:14.761060 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":237,"threads_started":1}
I20260620 02:22:14.762295 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:14.741692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:42211)
0620 02:22:14.742702 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:14.742764 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:14.744793 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:14.745630 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:14.745639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:14.746240 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:14.746905 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:14.746926 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.748112 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.748117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:14.749476 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:14.749491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.750379 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.750387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:14.750643 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:14.751436 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:14.751465 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:14.754027 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:14.758041 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:14.758050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:14.758068 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:14.758607 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:14.759032 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:14.759036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:14.759039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:14.759320 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:14.759952 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:14.759962 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:14.760456 (+ 494us) client_negotiation.cc:769] Sending connection context
0620 02:22:14.760706 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 02:22:14.761153 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":447,"threads_started":1}
I20260620 02:22:14.763898 18976 heartbeater.cc:347] Connected to a master server at 127.9.231.126:43409
I20260620 02:22:14.764238 18976 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:14.765122 18976 heartbeater.cc:511] Master 127.9.231.126:43409 requested a full tablet report, sending...
I20260620 02:22:14.766731 18516 ts_manager.cc:194] Registered new tserver with Master: 8e5a75adafa046c380d77b553d8c0920 (127.9.231.67:46095)
I20260620 02:22:14.767683 18516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:42211
I20260620 02:22:14.769913 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab.
Entry for principal oryx/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab.
WARNING: no policy specified for oryx/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab.
Entry for principal oryx/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab.
WARNING: no policy specified for oryx/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab.
Entry for principal oryx/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab.
I20260620 02:22:14.934509 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 18577
Jun 20 02:22:14 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:22:14.953984 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:14.935984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.9.231.65:45209)
0620 02:22:14.936736 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:14.936742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:14.936770 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:22:14.936833 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:14.936839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:14.937051 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:14.937183 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:14.938726 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.939954 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.941273 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:14.941910 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:14.942020 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:14.947355 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:14.947373 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:14.947391 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:14.947438 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:14.950619 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:14.951567 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:14.951576 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:14.951579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:14.951718 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:14.952215 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:14.952220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:14.952223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:14.952673 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:14.952991 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:14.953308 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 02:22:14.953528 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":453,"threads_started":1}
I20260620 02:22:14.955463 18701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:15.040915 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 18577
I20260620 02:22:15.058688 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 18712
Jun 20 02:22:15 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:22:15.077793 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:15.059794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.9.231.66:35053)
0620 02:22:15.060506 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:15.060513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:15.060542 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:22:15.060619 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:15.060626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:15.060889 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:15.061030 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:15.062464 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:15.063793 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:15.065266 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:15.065900 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:15.066034 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:15.070995 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:15.071014 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:15.071032 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:15.071151 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:15.074481 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:15.075427 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:15.075435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:15.075438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:15.075579 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:15.076016 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:15.076020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:15.076023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:15.076416 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:15.076663 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:15.077070 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 02:22:15.077440 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":344,"threads_started":1}
I20260620 02:22:15.079360 18836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:15.164902 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 18712
I20260620 02:22:15.181928 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 18847
Jun 20 02:22:15 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:22:15.200397 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:15.183149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.9.231.67:46095)
0620 02:22:15.183692 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:15.183699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:15.183729 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:22:15.183789 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:15.183795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:15.184046 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:15.184195 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:15.185697 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:15.186723 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:15.188224 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:15.188811 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:15.188929 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:15.193689 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:15.193706 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:15.193724 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:15.193791 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:15.197090 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:15.197937 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:15.197946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:15.197948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:15.198089 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:15.198487 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:15.198491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:15.198494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:15.198904 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:15.199260 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:15.199661 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 02:22:15.199895 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":268,"threads_started":1}
I20260620 02:22:15.201778 18971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:15.289515 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 18847
I20260620 02:22:15.306881 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 18483
I20260620 02:22:15.308320 18546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:15.419251 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 18483
I20260620 02:22:15.441252 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:43409
--webserver_interface=127.9.231.126
--webserver_port=33621
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:43409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:15.795082 18994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:15.795408 18994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:15.795588 18994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:15.803045 18994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:22:15.803196 18994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:15.803294 18994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:15.803385 18994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:22:15.803481 18994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:22:15.813838 18994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:43409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=33621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:15.815326 18994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:15.816728 18994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:15.830138 18999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:15.830052 19002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:15.830163 19000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:15.831163 18994 server_base.cc:1034] running on GCE node
I20260620 02:22:15.832033 18994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:15.833597 18994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:15.834839 18994 hybrid_clock.cc:648] HybridClock initialized: now 1781922135834791 us; error 33 us; skew 500 ppm
I20260620 02:22:15.835572 18994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:15 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922135, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:15.838964 18994 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:22:15.840992 18994 webserver.cc:460] Webserver started at http://127.9.231.126:33621/ using document root <none> and password file <none>
I20260620 02:22:15.841741 18994 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:15.841871 18994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:15.848348 18994 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:22:15.851739 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:15.853309 18994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:22:15.853526 18994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "f878be27a4154fe2a980355130f56179"
format_stamp: "Formatted at 2026-06-20 02:22:12 on dist-test-slave-9vg1"
server_key: "2d67ddeea89194e8808ab6793a3e8fe6"
server_key_iv: "253896c6d1e8f131e758e2eedcd1950c"
server_key_version: "encryptionkey@0"
I20260620 02:22:15.854686 18994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:15.873001 18994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:15.878171 18994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:15.878509 18994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:15.900347 18994 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:43409
I20260620 02:22:15.900368 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:43409 every 8 connection(s)
I20260620 02:22:15.902143 18994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:22:15.906471 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 18994
I20260620 02:22:15.907359 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:45209
--local_ip_for_outbound_sockets=127.9.231.65
--tserver_master_addrs=127.9.231.126:43409
--webserver_port=36927
--webserver_interface=127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:22:15.909637 19062 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:22:15.913141 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Bootstrap starting.
I20260620 02:22:15.942088 19062 log.cc:824] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:15.954077 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:22:15.954797 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Bootstrap complete.
I20260620 02:22:15.968806 19062 raft_consensus.cc:348] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:15.970293 19062 raft_consensus.cc:729] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f878be27a4154fe2a980355130f56179, State: Initialized, Role: FOLLOWER
I20260620 02:22:15.971141 19062 consensus_queue.cc:260] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:15.971457 19062 raft_consensus.cc:388] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:22:15.971582 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:22:15.971768 19062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:22:15.978237 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:15.979107 19062 leader_election.cc:302] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f878be27a4154fe2a980355130f56179; no voters:
I20260620 02:22:15.979905 19062 leader_election.cc:288] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:22:15.980154 19066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:22:15.981523 19066 raft_consensus.cc:686] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [term 2 LEADER]: Becoming Leader. State: Replica: f878be27a4154fe2a980355130f56179, State: Running, Role: LEADER
I20260620 02:22:15.982148 19066 consensus_queue.cc:237] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } }
I20260620 02:22:15.982285 19062 sys_catalog.cc:564] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:22:15.984614 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f878be27a4154fe2a980355130f56179. Latest consensus state: current_term: 2 leader_uuid: "f878be27a4154fe2a980355130f56179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } } }
I20260620 02:22:15.985056 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:15.985641 19067 sys_catalog.cc:455] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f878be27a4154fe2a980355130f56179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878be27a4154fe2a980355130f56179" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43409 } } }
I20260620 02:22:15.985931 19072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:22:15.985958 19067 sys_catalog.cc:458] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:15.994668 19072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:22:15.996273 19072 catalog_manager.cc:1259] Loaded cluster ID: 8cdc90b714ca4d76a4e04bb53a046ba4
I20260620 02:22:15.996589 19072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:22:16.001169 19072 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:22:16.002986 19072 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f878be27a4154fe2a980355130f56179: Loaded TSK: 0
I20260620 02:22:16.004935 19072 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:22:16.292207 19064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:16.292529 19064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:16.292680 19064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:16.300411 19064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:16.300547 19064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:16.300722 19064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:22:16.311223 19064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.65.keytab
--principal=oryx/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:45209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=36927
--tserver_master_addrs=127.9.231.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:16.312665 19064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:16.314055 19064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:16.329665 19089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:16.329994 19092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:16.331081 19090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:16.331763 19064 server_base.cc:1034] running on GCE node
I20260620 02:22:16.332579 19064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:16.333809 19064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:16.335083 19064 hybrid_clock.cc:648] HybridClock initialized: now 1781922136334989 us; error 52 us; skew 500 ppm
I20260620 02:22:16.335485 19064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:16 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922136, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:16.339785 19064 init.cc:377] Logged in from keytab as oryx/127.9.231.65@KRBTEST.COM (short username oryx)
I20260620 02:22:16.341903 19064 webserver.cc:460] Webserver started at http://127.9.231.65:36927/ using document root <none> and password file <none>
I20260620 02:22:16.342916 19064 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:16.343103 19064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:16.350785 19064 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 02:22:16.354523 19099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:16.356474 19064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:22:16.356688 19064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "4de3cee863d84a4697d5cd9c40845798"
format_stamp: "Formatted at 2026-06-20 02:22:13 on dist-test-slave-9vg1"
server_key: "7e32bf6efe00bdf60c7eefa47ea00d7a"
server_key_iv: "d2761d4515b7ecc0df8c5c8a46f4270c"
server_key_version: "encryptionkey@0"
I20260620 02:22:16.358227 19064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:16.380530 19064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:16.386222 19064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:16.386508 19064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:16.387645 19064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:16.389114 19064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:16.389207 19064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:16.389292 19064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:16.389360 19064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:16.425230 19064 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:45209
I20260620 02:22:16.425335 19211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:45209 every 8 connection(s)
I20260620 02:22:16.426853 19064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:22:16.428733 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 19064
I20260620 02:22:16.429534 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:35053
--local_ip_for_outbound_sockets=127.9.231.66
--tserver_master_addrs=127.9.231.126:43409
--webserver_port=36055
--webserver_interface=127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:22:16 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:16 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:22:16.822824 19216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:16.823251 19216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:16.823412 19216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:16.831247 19216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:16.831427 19216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:16.831597 19216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:22:16.843183 19216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.66.keytab
--principal=oryx/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:35053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=36055
--tserver_master_addrs=127.9.231.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:16.844720 19216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:16.846167 19216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:16.866379 19223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:16.867597 19222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:16.868252 19225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:16.869606 19216 server_base.cc:1034] running on GCE node
I20260620 02:22:16.870882 19216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:16.872833 19216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:16.874332 19216 hybrid_clock.cc:648] HybridClock initialized: now 1781922136874179 us; error 114 us; skew 500 ppm
I20260620 02:22:16.874966 19216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:16 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922136, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:16.880919 19216 init.cc:377] Logged in from keytab as oryx/127.9.231.66@KRBTEST.COM (short username oryx)
I20260620 02:22:16.882916 19216 webserver.cc:460] Webserver started at http://127.9.231.66:36055/ using document root <none> and password file <none>
I20260620 02:22:16.883890 19216 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:16.884060 19216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:16.891140 19216 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:22:16.895129 19232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:16.897171 19216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:22:16.897435 19216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "1d17f7e35d084c9a987f9c05ac3a1001"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "4ed6bbc9d4ae7b2d455644e40d3a75d8"
server_key_iv: "07be752d7b8eb313567d9bc42fe08fca"
server_key_version: "encryptionkey@0"
I20260620 02:22:16.898653 19216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:16.933214 19216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:16.938114 19216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:16.938447 19216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:16.939755 19216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:16.941373 19216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:16.941514 19216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:16.941638 19216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:16.941726 19216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:16.992904 19216 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:35053
I20260620 02:22:16.992923 19344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:35053 every 8 connection(s)
I20260620 02:22:16.994493 19216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:22:17.001403 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 19216
I20260620 02:22:17.002483 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:46095
--local_ip_for_outbound_sockets=127.9.231.67
--tserver_master_addrs=127.9.231.126:43409
--webserver_port=36407
--webserver_interface=127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:22:17.388154 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:17.388536 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:17.388726 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:17.397929 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:17.398104 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:17.398313 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:22:17.410401 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/krb5kdc/oryx_127.9.231.67.keytab
--principal=oryx/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:46095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=36407
--tserver_master_addrs=127.9.231.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:17.412063 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:17.413527 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:17.431951 19355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:17.439723 19356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:17.453594 19358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:17.455328 19349 server_base.cc:1034] running on GCE node
I20260620 02:22:17.456723 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:17.459022 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:17.460685 19349 hybrid_clock.cc:648] HybridClock initialized: now 1781922137460575 us; error 90 us; skew 500 ppm
I20260620 02:22:17.461486 19349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922137, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:17.469074 19349 init.cc:377] Logged in from keytab as oryx/127.9.231.67@KRBTEST.COM (short username oryx)
I20260620 02:22:17.472628 19349 webserver.cc:460] Webserver started at http://127.9.231.67:36407/ using document root <none> and password file <none>
I20260620 02:22:17.474169 19349 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:17.474419 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:17.486847 19349 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 02:22:17.493067 19365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:17.496306 19349 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260620 02:22:17.496750 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "8e5a75adafa046c380d77b553d8c0920"
format_stamp: "Formatted at 2026-06-20 02:22:14 on dist-test-slave-9vg1"
server_key: "6ca2fb7dee6b86ee58778e4bea900cf5"
server_key_iv: "e7f2443b5d8a8169bf4b85c01f8a32c6"
server_key_version: "encryptionkey@0"
I20260620 02:22:17.499243 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:17.533895 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:17.543843 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:17.544442 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:17.546540 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:17.549738 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:17.550063 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:22:17.550311 19349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:17.550477 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:17.629791 19349 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:46095
I20260620 02:22:17.629830 19477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:46095 every 8 connection(s)
I20260620 02:22:17.631651 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:22:17.633023 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 19349
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:17.692469 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:17.634491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.9.231.126:43409)
0620 02:22:17.652832 (+ 18341us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:17.652842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:17.663073 (+ 10231us) server_negotiation.cc:407] Connection header received
0620 02:22:17.663326 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:17.663349 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:17.663447 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:17.663569 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:17.668719 (+ 5150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.670156 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.674539 (+ 4383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.674998 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.675210 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.681284 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:17.681300 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:17.681328 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:17.681389 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:17.686334 (+ 4945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:17.687643 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:17.687657 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:17.687670 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:17.687941 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:17.688484 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:17.688493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:17.688499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:17.689297 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:17.689646 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:17.690551 (+ 905us) server_negotiation.cc:299] Negotiation successful
0620 02:22:17.691186 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18102,"thread_start_us":18367,"threads_started":1}
W20260620 02:22:17.914846 19215 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.470s user 0.002s sys 0.012s
W20260620 02:22:17.917194 19214 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:16.430247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:48713 (local address 127.9.231.126:43409)
0620 02:22:16.432359 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:16.432378 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:16.432445 (+ 67us) server_negotiation.cc:407] Connection header received
0620 02:22:16.434736 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:16.434751 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:16.435224 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:16.435573 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:16.437756 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:16.439683 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:16.442505 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:16.443328 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:16.443494 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.916687 (+1473193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:48713: BlockingRecv error: recv got EOF from 127.9.231.65:48713 (error 108)
Metrics: {"server-negotiator.queue_time_us":1796,"thread_start_us":2126,"threads_started":1}
I20260620 02:22:17.917997 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:16.430236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:48713)
0620 02:22:16.431243 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:16.431308 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:16.434066 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:16.435799 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:16.435813 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:16.436636 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:16.437479 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:16.437511 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:16.439899 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:16.439908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:16.442299 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:16.442325 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:16.443613 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:16.443623 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:16.443997 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:16.444842 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:16.444880 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:17.916219 (+1471339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":372,"threads_started":1}
W20260620 02:22:17.919620 19212 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:43409 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.231.126:43409: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:17.948649 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:17.921537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:56703)
0620 02:22:17.922608 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:17.922639 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:17.922909 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:17.926079 (+ 3170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:17.926088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:17.926734 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:17.927050 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.927097 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.929501 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.929510 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.931978 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.932002 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.933681 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.933692 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.933966 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.940987 (+ 7021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:17.941031 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:17.948140 (+ 7109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":859}
W20260620 02:22:17.949012 19214 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:17.922049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:56703 (local address 127.9.231.126:43409)
0620 02:22:17.922331 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:17.922357 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:17.922749 (+ 392us) server_negotiation.cc:407] Connection header received
0620 02:22:17.923176 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:17.923187 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:17.923310 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:17.925763 (+ 2453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:17.927388 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.929181 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.932225 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.933371 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.933594 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.948576 (+ 14982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:56703: BlockingRecv error: recv got EOF from 127.9.231.65:56703 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:17 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:17.975023 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:17.950232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:50965)
0620 02:22:17.951459 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:17.951491 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:17.951683 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:17.952549 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:17.952556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:17.953312 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:17.953659 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.953689 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.959395 (+ 5706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.959406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.962375 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.962408 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.964687 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.964698 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.965012 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.966209 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:17.966256 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:17.974498 (+ 8242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":15692,"threads_started":1}
W20260620 02:22:17.975350 19214 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:17.950370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:50965 (local address 127.9.231.126:43409)
0620 02:22:17.950609 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:17.950618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:17.951851 (+ 1233us) server_negotiation.cc:407] Connection header received
0620 02:22:17.951954 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:17.951963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:17.952100 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:17.952251 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:17.954723 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.956532 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.962668 (+ 6136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.963862 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.964101 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.974857 (+ 10756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:50965: BlockingRecv error: recv got EOF from 127.9.231.65:50965 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 02:22:17.975888 19212 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:22:18.419308 19348 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.410s user 0.003s sys 0.014s
W20260620 02:22:18.421123 19347 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:16.997653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:55725 (local address 127.9.231.126:43409)
0620 02:22:16.998119 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:16.998123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:16.999804 (+ 1681us) server_negotiation.cc:407] Connection header received
0620 02:22:17.002077 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:17.002083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:17.002183 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:17.002303 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:17.004387 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.005398 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.007126 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.007558 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.007699 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.420875 (+1413176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:55725: BlockingRecv error: recv got EOF from 127.9.231.66:55725 (error 108)
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":260,"threads_started":1}
I20260620 02:22:18.422403 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:16.998608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:55725)
0620 02:22:16.999622 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:16.999664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:17.001760 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:17.002645 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:17.002662 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:17.003405 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:17.004153 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.004192 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.005582 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.005588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.006950 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.006963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.007810 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.007820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.008097 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.008826 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:17.008851 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:18.420694 (+1411843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":473,"threads_started":1}
W20260620 02:22:18.423488 19345 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:43409 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.231.126:43409: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:22:18.437804 19347 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:18.424344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:52607 (local address 127.9.231.126:43409)
0620 02:22:18.424514 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:18.424518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:18.424638 (+ 120us) server_negotiation.cc:407] Connection header received
0620 02:22:18.424954 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:18.424960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:18.425043 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:18.425166 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:18.426582 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.427711 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.430576 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.431218 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.431561 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.437623 (+ 6062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:52607: BlockingRecv error: recv got EOF from 127.9.231.66:52607 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:22:18.437911 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:18.424184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:52607)
0620 02:22:18.424512 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:18.424544 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:18.424769 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:18.425275 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:18.425283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:18.426024 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:18.426352 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.426371 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.427939 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.427948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.430372 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.430399 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.431471 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.431482 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.431768 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.432675 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:18.432705 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:18.437421 (+ 4716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:22:18.452183 19347 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:18.439115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:48973 (local address 127.9.231.126:43409)
0620 02:22:18.439358 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:18.439367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:18.439395 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:22:18.439485 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:18.439494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:18.439624 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:18.439776 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:18.440480 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.442223 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.443905 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.444998 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.445186 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.451972 (+ 6786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:48973: BlockingRecv error: recv got EOF from 127.9.231.66:48973 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:22:18.452162 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:18.438766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:48973)
0620 02:22:18.438934 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:18.438958 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:18.439107 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:18.439821 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:18.439824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:18.440154 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:18.440317 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.440325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.442430 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.442434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.443643 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.443657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.445187 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.445193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.445372 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.446096 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:18.446135 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:18.451680 (+ 5545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:22:18.452672 19345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:22:18.840795 19482 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.161s user 0.008s sys 0.011s
W20260620 02:22:18.841921 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:17.637562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:53743 (local address 127.9.231.126:43409)
0620 02:22:17.657627 (+ 20065us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:17.657637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:17.659887 (+ 2250us) server_negotiation.cc:407] Connection header received
0620 02:22:17.669461 (+ 9574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:17.669468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:17.669562 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:17.669663 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:17.672473 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.674005 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.676832 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.677521 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.678183 (+ 662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.841717 (+1163534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:53743: BlockingRecv error: recv got EOF from 127.9.231.67:53743 (error 108)
Metrics: {"server-negotiator.queue_time_us":19875,"thread_start_us":20070,"threads_started":1}
I20260620 02:22:18.842561 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:17.636721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:53743)
0620 02:22:17.658670 (+ 21949us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:17.658766 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:17.665313 (+ 6547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:17.670760 (+ 5447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:17.670776 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:17.671522 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:17.672202 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.672223 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.674269 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.674276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.676646 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:17.676662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:17.678549 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:17.678559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:17.678967 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:17.679898 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:17.679936 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:18.841535 (+1161599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21391,"thread_start_us":24046,"threads_started":1}
W20260620 02:22:18.843387 19478 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:43409 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.231.126:43409: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:18.853046 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:18.844082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:57941)
0620 02:22:18.844283 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:18.844299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:18.844396 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:18.844688 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:18.844691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:18.845049 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:18.845219 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.845227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.846202 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.846206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.847244 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.847256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.847951 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.847955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.848098 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.848708 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:18.848725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:18.852672 (+ 3947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 02:22:18.853093 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:18.844163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:57941 (local address 127.9.231.126:43409)
0620 02:22:18.844323 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:18.844327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:18.844345 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:22:18.844511 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:18.844514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:18.844573 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:18.844651 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:18.845337 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.846080 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.847377 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.847764 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.847857 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.852875 (+ 5018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:57941: BlockingRecv error: recv got EOF from 127.9.231.67:57941 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:18.869165 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:18.853908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:38999)
0620 02:22:18.854071 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:18.854091 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:18.854191 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:18.854537 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:18.854540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:18.854873 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:18.855037 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.855046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.856098 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.856102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.857072 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.857084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.857790 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.857794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.858000 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.864552 (+ 6552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:18.864580 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:18.868957 (+ 4377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:22:18.869570 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:18.854047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:38999 (local address 127.9.231.126:43409)
0620 02:22:18.854271 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:18.854275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:18.854290 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:22:18.854335 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:18.854338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:18.854410 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:18.854489 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:18.855206 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.855963 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.857204 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.857651 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.857745 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.869221 (+ 11476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:38999: BlockingRecv error: recv got EOF from 127.9.231.67:38999 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260620 02:22:18.869735 19478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:18 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:18.986578 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:18.976633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:55053)
0620 02:22:18.977345 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:18.977361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:18.977491 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:18.977870 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:18.977874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:18.978298 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:18.978452 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.978461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.979582 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.979585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.980865 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:18.980879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.981796 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.981802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:18.981989 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.982700 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:18.982718 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:18.986331 (+ 3613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":476,"threads_started":1}
W20260620 02:22:18.986861 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:18.976758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:55053 (local address 127.9.231.126:43409)
0620 02:22:18.976980 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:18.976985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:18.977473 (+ 488us) server_negotiation.cc:407] Connection header received
0620 02:22:18.977603 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:18.977607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:18.977710 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:18.977798 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:18.978596 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.979439 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.981036 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:18.981614 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:18.981711 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:18.986569 (+ 4858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:55053: BlockingRecv error: recv got EOF from 127.9.231.65:55053 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 20 02:22:19 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:19 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:19.463280 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:19.453324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:48355)
0620 02:22:19.453977 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:19.453993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:19.454161 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:19.454508 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:19.454512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:19.454963 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:19.455172 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:19.455183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.456235 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.456242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:19.457227 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:19.457240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.458007 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.458013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:19.458178 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:19.458872 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:19.458890 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:19.462864 (+ 3974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":446,"threads_started":1}
W20260620 02:22:19.463482 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:19.453432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:48355 (local address 127.9.231.126:43409)
0620 02:22:19.453700 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:19.453708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:19.454079 (+ 371us) server_negotiation.cc:407] Connection header received
0620 02:22:19.454291 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:19.454296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:19.454364 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:19.454488 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:19.455330 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.456090 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.457379 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.457819 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.457956 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:19.463177 (+ 5221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:48355: BlockingRecv error: recv got EOF from 127.9.231.66:48355 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 02:22:19 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:19 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:19.880246 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:19.870342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:32931)
0620 02:22:19.870907 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:19.870924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:19.871040 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:19.871380 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:19.871383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:19.871855 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:19.872002 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:19.872010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.873028 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.873032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:19.874051 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:19.874064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.874847 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.874853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:19.874974 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:19.875660 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:19.875677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:19.879882 (+ 4205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":290,"threads_started":1}
W20260620 02:22:19.880399 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:19.870446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:32931 (local address 127.9.231.126:43409)
0620 02:22:19.870725 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:19.870731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:19.871025 (+ 294us) server_negotiation.cc:407] Connection header received
0620 02:22:19.871174 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:19.871178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:19.871259 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:19.871350 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:19.872124 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.872866 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.874196 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.874639 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.874737 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:19.880144 (+ 5407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:32931: BlockingRecv error: recv got EOF from 127.9.231.67:32931 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 02:22:19 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:19 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:19.997545 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:19.987717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:56709)
0620 02:22:19.988354 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:19.988372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:19.988510 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:19.988871 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:19.988876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:19.989276 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:19.989476 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:19.989486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.990672 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.990680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:19.991688 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:19.991701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.992412 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.992418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:19.992608 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:19.993271 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:19.993289 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:19.997159 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":380,"threads_started":1}
W20260620 02:22:19.997717 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:19.987896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:56709 (local address 127.9.231.126:43409)
0620 02:22:19.988090 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:19.988094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:19.988453 (+ 359us) server_negotiation.cc:407] Connection header received
0620 02:22:19.988620 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:19.988624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:19.988688 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:19.988782 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:19.989663 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.990530 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.991845 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:19.992240 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:19.992390 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:19.997426 (+ 5036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:56709: BlockingRecv error: recv got EOF from 127.9.231.65:56709 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 02:22:20 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:20 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:20.474499 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:20.464466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:45849)
0620 02:22:20.465056 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:20.465073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:20.465193 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:20.465544 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:20.465548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:20.465959 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:20.466142 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:20.466151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.467178 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.467183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:20.468170 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:20.468184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.469092 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.469098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:20.469256 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:20.469883 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:20.469903 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:20.474147 (+ 4244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":370,"threads_started":1}
W20260620 02:22:20.474606 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:20.464528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:45849 (local address 127.9.231.126:43409)
0620 02:22:20.464718 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:20.464723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:20.465152 (+ 429us) server_negotiation.cc:407] Connection header received
0620 02:22:20.465337 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:20.465341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:20.465406 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:20.465495 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:20.466280 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.467029 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.468356 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.468887 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.469021 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:20.474380 (+ 5359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:45849: BlockingRecv error: recv got EOF from 127.9.231.66:45849 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 02:22:20 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:20 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:20.891494 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:20.881428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:45899)
0620 02:22:20.882290 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:20.882313 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:20.882480 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:20.882985 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:20.882992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:20.883568 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:20.883765 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:20.883775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.884821 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.884825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:20.885732 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:20.885742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.886452 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.886457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:20.886604 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:20.887356 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:20.887374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:20.891238 (+ 3864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":373,"threads_started":1}
W20260620 02:22:20.891676 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:20.881648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:45899 (local address 127.9.231.126:43409)
0620 02:22:20.881961 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:20.881975 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:20.882414 (+ 439us) server_negotiation.cc:407] Connection header received
0620 02:22:20.882658 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:20.882665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:20.882766 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:20.882932 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:20.883907 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.884679 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.885907 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:20.886307 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:20.886408 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:20.891428 (+ 5020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:45899: BlockingRecv error: recv got EOF from 127.9.231.67:45899 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
Jun 20 02:22:21 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:21 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:21.008457 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:20.998726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:47839)
0620 02:22:20.999301 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:20.999318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:20.999477 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:20.999827 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:20.999833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:21.000240 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:21.000413 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:21.000422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.001517 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.001523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:21.002485 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:21.002499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.003347 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.003354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:21.003561 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:21.004197 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:21.004215 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:21.008209 (+ 3994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":328,"threads_started":1}
W20260620 02:22:21.008657 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:20.998862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:47839 (local address 127.9.231.126:43409)
0620 02:22:20.999138 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:20.999145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:20.999429 (+ 284us) server_negotiation.cc:407] Connection header received
0620 02:22:20.999614 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:20.999619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:20.999688 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:20.999792 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:21.000567 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.001340 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.002656 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.003183 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.003315 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:21.008396 (+ 5081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:47839: BlockingRecv error: recv got EOF from 127.9.231.65:47839 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 02:22:21 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:21 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:21.485538 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:21.475566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:53437)
0620 02:22:21.476086 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:21.476103 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:21.476216 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:21.476612 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:21.476616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:21.477131 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:21.477311 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:21.477321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.478426 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.478430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:21.479421 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:21.479433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.480145 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.480149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:21.480286 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:21.481084 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:21.481104 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:21.485277 (+ 4173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":276,"threads_started":1}
W20260620 02:22:21.486006 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:21.475604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:53437 (local address 127.9.231.126:43409)
0620 02:22:21.475786 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:21.475797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:21.476191 (+ 394us) server_negotiation.cc:407] Connection header received
0620 02:22:21.476396 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:21.476400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:21.476465 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:21.476550 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:21.477495 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.478281 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.479556 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.480011 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.480121 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:21.485574 (+ 5453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:53437: BlockingRecv error: recv got EOF from 127.9.231.66:53437 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 02:22:21 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:21 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:21.902133 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:21.892448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:56679)
0620 02:22:21.892995 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:21.893015 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:21.893141 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:21.893465 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:21.893468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:21.893811 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:21.893973 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:21.893982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.895142 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.895146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:21.896048 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:21.896060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.896813 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.896819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:21.896953 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:21.897638 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:21.897657 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:21.901785 (+ 4128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":320,"threads_started":1}
W20260620 02:22:21.902469 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:21.892621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:56679 (local address 127.9.231.126:43409)
0620 02:22:21.892909 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:21.892915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:21.893120 (+ 205us) server_negotiation.cc:407] Connection header received
0620 02:22:21.893256 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:21.893260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:21.893335 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:21.893418 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:21.894141 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.894930 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.896229 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:21.896665 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:21.896793 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:21.902223 (+ 5430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:56679: BlockingRecv error: recv got EOF from 127.9.231.67:56679 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Jun 20 02:22:22 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:22 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:22.020057 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:22.009830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:56825)
0620 02:22:22.010395 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:22.010411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:22.010572 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:22.011002 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:22.011007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:22.011465 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:22.011628 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:22.011637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.012693 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.012700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:22.013710 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:22.013722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.014411 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.014417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:22.014597 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:22.015370 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:22.015389 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:22.019615 (+ 4226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":283,"threads_started":1}
W20260620 02:22:22.020112 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:22.010001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:56825 (local address 127.9.231.126:43409)
0620 02:22:22.010240 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:22.010246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:22.010500 (+ 254us) server_negotiation.cc:407] Connection header received
0620 02:22:22.010716 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:22.010720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:22.010784 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:22.010966 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:22.011799 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.012557 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.013868 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.014260 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.014397 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:22.019847 (+ 5450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:56825: BlockingRecv error: recv got EOF from 127.9.231.65:56825 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 02:22:22 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:22 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:22:22.496860 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:22.486676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:54813 (local address 127.9.231.126:43409)
0620 02:22:22.486980 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:22.486988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:22.487372 (+ 384us) server_negotiation.cc:407] Connection header received
0620 02:22:22.487627 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:22.487633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:22.487714 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:22.487822 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:22.488632 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.489445 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.490900 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.491479 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.491627 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:22.496648 (+ 5021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:54813: BlockingRecv error: recv got EOF from 127.9.231.66:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260620 02:22:22.496879 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:22.486555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:54813)
0620 02:22:22.487251 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:22.487270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:22.487473 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:22.487852 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:22.487857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:22.488300 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:22.488473 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:22.488482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.489588 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.489595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:22.490736 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:22.490748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.491639 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.491645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:22.491850 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:22.492487 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:22.492506 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:22.496456 (+ 3950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":317,"threads_started":1}
Jun 20 02:22:22 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:22 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:22.914214 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:22.903851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:46117)
0620 02:22:22.904541 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:22.904558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:22.904683 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:22.905209 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:22.905215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:22.905742 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:22.905951 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:22.905973 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.907404 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.907408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:22.908495 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:22.908506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.909453 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.909457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:22.909583 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:22.910283 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:22.910300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:22.913958 (+ 3658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":409,"threads_started":1}
W20260620 02:22:22.914475 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:22.903872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:46117 (local address 127.9.231.126:43409)
0620 02:22:22.904084 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:22.904090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:22.904645 (+ 555us) server_negotiation.cc:407] Connection header received
0620 02:22:22.904839 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:22.904846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:22.904953 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:22.905064 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:22.906133 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.907221 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.908694 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:22.909297 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:22.909414 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:22.914193 (+ 4779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:46117: BlockingRecv error: recv got EOF from 127.9.231.67:46117 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 02:22:23 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:23 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:23.038071 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:23.021228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:46109)
0620 02:22:23.021730 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:23.021745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:23.021859 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:23.022255 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:23.022259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:23.022711 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:23.022865 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:23.022874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.023953 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.023957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:23.024908 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:23.024920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.025561 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.025565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:23.025698 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:23.034053 (+ 8355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:23.034081 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:23.037838 (+ 3757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":278,"threads_started":1}
W20260620 02:22:23.038393 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:23.021295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:46109 (local address 127.9.231.126:43409)
0620 02:22:23.021499 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:23.021504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:23.021853 (+ 349us) server_negotiation.cc:407] Connection header received
0620 02:22:23.021980 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:23.021984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:23.022046 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:23.022141 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:23.023010 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.023826 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.025048 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.025435 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.025526 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:23.038075 (+ 12549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:46109: BlockingRecv error: recv got EOF from 127.9.231.65:46109 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 02:22:23 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:23 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:23.507890 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:23.497851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:39675)
0620 02:22:23.498440 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:23.498460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:23.498594 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:23.498990 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:23.498995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:23.499571 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:23.499830 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:23.499843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.501043 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.501048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:23.501991 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:23.502003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.502726 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.502730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:23.502931 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:23.503694 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:23.503723 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:23.507515 (+ 3792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":331,"threads_started":1}
W20260620 02:22:23.508107 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:23.498083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:39675 (local address 127.9.231.126:43409)
0620 02:22:23.498256 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:23.498262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:23.498575 (+ 313us) server_negotiation.cc:407] Connection header received
0620 02:22:23.498722 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:23.498726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:23.498839 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:23.498929 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:23.499985 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.500882 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.502122 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.502594 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.502697 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:23.507763 (+ 5066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:39675: BlockingRecv error: recv got EOF from 127.9.231.66:39675 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 02:22:23 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:23 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:23.926077 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:23.915345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:41755)
0620 02:22:23.915843 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:23.915861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:23.915985 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:23.916364 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:23.916368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:23.916846 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:23.916998 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:23.917007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.918172 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.918177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:23.919333 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:23.919347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.920308 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.920314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:23.920461 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:23.921232 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:23.921275 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:23.925849 (+ 4574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
W20260620 02:22:23.926345 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:23.915418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:41755 (local address 127.9.231.126:43409)
0620 02:22:23.915671 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:23.915678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:23.915950 (+ 272us) server_negotiation.cc:407] Connection header received
0620 02:22:23.916107 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:23.916111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:23.916190 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:23.916283 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:23.917154 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.918007 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.919505 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:23.919957 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:23.920086 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:23.926086 (+ 6000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:41755: BlockingRecv error: recv got EOF from 127.9.231.67:41755 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 02:22:24 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:24 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:24.049259 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:24.039110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:55495)
0620 02:22:24.039693 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:24.039712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:24.039903 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:24.040301 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:24.040308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:24.040833 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:24.041022 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:24.041033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.042267 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.042275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:24.043418 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:24.043431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.044333 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.044341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:24.044589 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:24.045247 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:24.045265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:24.048939 (+ 3674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":292,"threads_started":1}
W20260620 02:22:24.049521 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:24.039171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:55495 (local address 127.9.231.126:43409)
0620 02:22:24.039517 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:24.039525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:24.039818 (+ 293us) server_negotiation.cc:407] Connection header received
0620 02:22:24.040055 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:24.040064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:24.040142 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:24.040269 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:24.041239 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.042086 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.043592 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.044128 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.044306 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:24.049202 (+ 4896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:55495: BlockingRecv error: recv got EOF from 127.9.231.65:55495 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Jun 20 02:22:24 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:24 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:24.518823 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:24.508951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:53411)
0620 02:22:24.509539 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:24.509564 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:24.509746 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:24.510203 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:24.510211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:24.510725 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:24.510922 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:24.510936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.512135 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.512139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:24.513101 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:24.513111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.513901 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.513906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:24.514032 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:24.514690 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:24.514722 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:24.518569 (+ 3847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"spinlock_wait_cycles":50560,"thread_start_us":426,"threads_started":1}
W20260620 02:22:24.519106 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:24.509077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:53411 (local address 127.9.231.126:43409)
0620 02:22:24.509360 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:24.509369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:24.509714 (+ 345us) server_negotiation.cc:407] Connection header received
0620 02:22:24.509917 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:24.509923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:24.510013 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:24.510128 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:24.511110 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.511974 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.513252 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.513752 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.513867 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:24.518832 (+ 4965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:53411: BlockingRecv error: recv got EOF from 127.9.231.66:53411 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 20 02:22:24 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:24 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:24.937260 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:24.927210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:39821)
0620 02:22:24.927766 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:24.927790 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:24.927928 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:24.928284 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:24.928288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:24.928952 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:24.929145 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:24.929157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.930241 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.930245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:24.931352 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:24.931367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.932107 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.932112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:24.932277 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:24.932931 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:24.932949 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:24.937015 (+ 4066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":264,"threads_started":1}
W20260620 02:22:24.937565 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:24.927711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:39821 (local address 127.9.231.126:43409)
0620 02:22:24.927887 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:24.927892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:24.927911 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:22:24.928036 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:24.928040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:24.928116 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:24.928199 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:24.929310 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.930103 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.931504 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:24.931946 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:24.932045 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:24.937228 (+ 5183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:39821: BlockingRecv error: recv got EOF from 127.9.231.67:39821 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 02:22:25 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:25 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:25.061285 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:25.050462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:60209)
0620 02:22:25.051091 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:25.051108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:25.051233 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:25.051516 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:25.051520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:25.052031 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:25.052328 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:25.052339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.053421 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.053426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:25.054494 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:25.054506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.055425 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.055434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:25.055625 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:25.056493 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:25.056515 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:25.060906 (+ 4391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":299,"threads_started":1}
W20260620 02:22:25.061342 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:25.050568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:60209 (local address 127.9.231.126:43409)
0620 02:22:25.050734 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:25.050738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:25.051177 (+ 439us) server_negotiation.cc:407] Connection header received
0620 02:22:25.051319 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:25.051323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:25.051387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:25.051466 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:25.052478 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.053252 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.054636 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.055162 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.055290 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:25.061121 (+ 5831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:60209: BlockingRecv error: recv got EOF from 127.9.231.65:60209 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 02:22:25 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:25 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:25.529804 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:25.519966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:55701)
0620 02:22:25.520607 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:25.520625 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:25.520853 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:25.521313 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:25.521319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:25.521796 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:25.521988 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:25.521998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.523143 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.523150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:25.524208 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:25.524222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.525042 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.525048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:25.525233 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:25.525898 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:25.525917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:25.529547 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":287,"threads_started":1}
W20260620 02:22:25.530092 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:25.520100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:55701 (local address 127.9.231.126:43409)
0620 02:22:25.520310 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:25.520316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:25.520777 (+ 461us) server_negotiation.cc:407] Connection header received
0620 02:22:25.521001 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:25.521009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:25.521081 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:25.521194 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:25.522155 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.522957 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.524370 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.524861 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.525008 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:25.529778 (+ 4770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:55701: BlockingRecv error: recv got EOF from 127.9.231.66:55701 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 02:22:25 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:25 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:25.947762 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:25.938337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:34393)
0620 02:22:25.939029 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:25.939046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:25.939253 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:25.939614 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:25.939619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:25.940089 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:25.940283 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:25.940292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.941310 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.941316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:25.942273 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:25.942284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.942996 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.943002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:25.943201 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:25.943869 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:25.943886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:25.947408 (+ 3522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":341,"threads_started":1}
W20260620 02:22:25.947922 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:25.938448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:34393 (local address 127.9.231.126:43409)
0620 02:22:25.938882 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:25.938890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:25.939165 (+ 275us) server_negotiation.cc:407] Connection header received
0620 02:22:25.939396 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:25.939403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:25.939469 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:25.939581 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:25.940434 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.941161 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.942419 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:25.942840 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:25.942975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:25.947630 (+ 4655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:34393: BlockingRecv error: recv got EOF from 127.9.231.67:34393 (error 108)
Metrics: {"server-negotiator.queue_time_us":196}
Jun 20 02:22:26 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:26 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:26.072276 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:26.062332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:44863)
0620 02:22:26.062922 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:26.062942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:26.063155 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:26.063508 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:26.063512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:26.064039 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:26.064234 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:26.064243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.065299 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.065306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:26.066254 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:26.066266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.067020 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.067026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:26.067225 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:26.067888 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:26.067908 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:26.071876 (+ 3968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":289,"threads_started":1}
W20260620 02:22:26.072391 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:26.062583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:44863 (local address 127.9.231.126:43409)
0620 02:22:26.062787 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:26.062793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:26.063037 (+ 244us) server_negotiation.cc:407] Connection header received
0620 02:22:26.063283 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:26.063287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:26.063347 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:26.063472 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:26.064398 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.065130 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.066415 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.066837 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.066975 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:26.072082 (+ 5107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:44863: BlockingRecv error: recv got EOF from 127.9.231.65:44863 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 02:22:26 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:26 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:26.540602 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:26.530840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:49835)
0620 02:22:26.531372 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:26.531389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:26.531519 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:26.531972 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:26.531976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:26.532419 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:26.532646 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:26.532656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.534294 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.534298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:26.535268 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:26.535282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.536066 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.536070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:26.536182 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:26.536774 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:26.536793 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:26.540353 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
W20260620 02:22:26.541043 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:26.531097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:49835 (local address 127.9.231.126:43409)
0620 02:22:26.531310 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:26.531317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:26.531478 (+ 161us) server_negotiation.cc:407] Connection header received
0620 02:22:26.531750 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:26.531754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:26.531824 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:26.531913 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:26.532806 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.534145 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.535428 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.535892 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.535999 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:26.540619 (+ 4620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:49835: BlockingRecv error: recv got EOF from 127.9.231.66:49835 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 02:22:26 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:26 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:26.958878 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:26.948984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:39511)
0620 02:22:26.949555 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:26.949572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:26.949687 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:26.950020 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:26.950024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:26.950476 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:26.950653 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:26.950662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.951875 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.951880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:26.952872 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:26.952884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.953861 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.953868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:26.954044 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:26.954741 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:26.954760 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:26.958601 (+ 3841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":312,"threads_started":1}
W20260620 02:22:26.959229 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:26.949067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:39511 (local address 127.9.231.126:43409)
0620 02:22:26.949297 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:26.949301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:26.949666 (+ 365us) server_negotiation.cc:407] Connection header received
0620 02:22:26.949802 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:26.949806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:26.949875 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:26.949967 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:26.950825 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.951722 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.953046 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:26.953659 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:26.953822 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:26.958909 (+ 5087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:39511: BlockingRecv error: recv got EOF from 127.9.231.67:39511 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 02:22:27 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:27 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:27.083374 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:27.073450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:33469)
0620 02:22:27.074024 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:27.074041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:27.074166 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:27.074467 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:27.074470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:27.074920 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:27.075141 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:27.075152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.076175 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.076179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:27.077217 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:27.077229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.077908 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.077912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:27.078034 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:27.078812 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:27.078834 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:27.083102 (+ 4268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":359,"threads_started":1}
W20260620 02:22:27.083523 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:27.073501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:33469 (local address 127.9.231.126:43409)
0620 02:22:27.073756 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:27.073766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:27.074144 (+ 378us) server_negotiation.cc:407] Connection header received
0620 02:22:27.074281 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:27.074285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:27.074347 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:27.074432 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:27.075282 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.076044 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.077350 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.077754 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.077862 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:27.083284 (+ 5422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:33469: BlockingRecv error: recv got EOF from 127.9.231.65:33469 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 02:22:27 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:27 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:27.551383 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:27.541654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:44733)
0620 02:22:27.542329 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:27.542355 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:27.542513 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:27.542874 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:27.542879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:27.543338 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:27.543483 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:27.543493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.544598 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.544604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:27.545555 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:27.545567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.546341 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.546347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:27.546527 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:27.547184 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:27.547206 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:27.551014 (+ 3808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":394,"threads_started":1}
W20260620 02:22:27.551545 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:27.541821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:44733 (local address 127.9.231.126:43409)
0620 02:22:27.542043 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:27.542050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:27.542441 (+ 391us) server_negotiation.cc:407] Connection header received
0620 02:22:27.542649 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:27.542654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:27.542734 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:27.542840 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:27.543629 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.544449 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.545699 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.546158 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.546303 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:27.551275 (+ 4972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:44733: BlockingRecv error: recv got EOF from 127.9.231.66:44733 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 02:22:27 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:27 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:27.969859 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:27.959913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:53771)
0620 02:22:27.960459 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:27.960476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:27.960582 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:27.960955 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:27.960958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:27.961358 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:27.961503 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:27.961512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.962635 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.962639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:27.964314 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:27.964334 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.965328 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.965333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:27.965519 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:27.966107 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:27.966124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:27.969543 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":286,"threads_started":1}
W20260620 02:22:27.970003 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:27.960038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:53771 (local address 127.9.231.126:43409)
0620 02:22:27.960237 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:27.960241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:27.960571 (+ 330us) server_negotiation.cc:407] Connection header received
0620 02:22:27.960742 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:27.960746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:27.960812 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:27.960925 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:27.961708 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.962510 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.964515 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:27.965169 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:27.965263 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:27.969794 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:53771: BlockingRecv error: recv got EOF from 127.9.231.67:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 02:22:28 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:28 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:28.098372 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:28.084337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:35213)
0620 02:22:28.084938 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:28.084955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:28.085113 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:28.085476 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:28.085481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:28.085865 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:28.086019 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:28.086028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.087172 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.087179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:28.088334 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:28.088347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.089042 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.089048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:28.089218 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:28.094331 (+ 5113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:28.094350 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:28.098120 (+ 3770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":265,"threads_started":1}
W20260620 02:22:28.098654 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:28.084550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:35213 (local address 127.9.231.126:43409)
0620 02:22:28.084789 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:28.084796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:28.085038 (+ 242us) server_negotiation.cc:407] Connection header received
0620 02:22:28.085273 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:28.085277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:28.085342 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:28.085446 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:28.086161 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.086998 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.088470 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.088892 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.089026 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:28.098350 (+ 9324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:35213: BlockingRecv error: recv got EOF from 127.9.231.65:35213 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 02:22:28 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:28 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:28.561493 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:28.552430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:37393)
0620 02:22:28.552971 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:28.552988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:28.553101 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:28.553476 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:28.553479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:28.553879 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:28.554018 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:28.554027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.554988 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.554992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:28.556060 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:28.556072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.556756 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.556763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:28.556891 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:28.557549 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:28.557567 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:28.561245 (+ 3678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":322,"threads_started":1}
W20260620 02:22:28.561720 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:28.552524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:37393 (local address 127.9.231.126:43409)
0620 02:22:28.552792 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:28.552800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:28.553118 (+ 318us) server_negotiation.cc:407] Connection header received
0620 02:22:28.553210 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:28.553213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:28.553296 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:28.553408 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:28.554148 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.554851 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.556195 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.556592 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.556685 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:28.561459 (+ 4774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:37393: BlockingRecv error: recv got EOF from 127.9.231.66:37393 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 02:22:28 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:28 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:28.980556 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:28.970931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:57851)
0620 02:22:28.971554 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:28.971570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:28.971744 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:28.972107 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:28.972112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:28.972565 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:28.972746 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:28.972757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.974012 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.974017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:28.975090 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:28.975103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.975865 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.975887 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:28.976051 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:28.976695 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:28.976714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:28.980318 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":385,"threads_started":1}
W20260620 02:22:28.980834 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:28.971042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:57851 (local address 127.9.231.126:43409)
0620 02:22:28.971321 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:28.971330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:28.971675 (+ 345us) server_negotiation.cc:407] Connection header received
0620 02:22:28.971884 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:28.971889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:28.971965 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:28.972071 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:28.972899 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.973867 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.975250 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:28.975704 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:28.975835 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:28.980532 (+ 4697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:57851: BlockingRecv error: recv got EOF from 127.9.231.67:57851 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Jun 20 02:22:29 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:29 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:29.109699 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:29.099505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:53339)
0620 02:22:29.100091 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:29.100110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:29.100310 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:29.100688 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:29.100693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:29.101181 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:29.101370 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:29.101380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.102495 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.102504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:29.103781 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:29.103795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.104715 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.104722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:29.104922 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:29.105577 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:29.105596 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:29.109328 (+ 3732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":334,"threads_started":1}
W20260620 02:22:29.109869 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:29.099543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:53339 (local address 127.9.231.126:43409)
0620 02:22:29.099891 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:29.099900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:29.100225 (+ 325us) server_negotiation.cc:407] Connection header received
0620 02:22:29.100463 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:29.100468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:29.100536 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:29.100648 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:29.101536 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.102343 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.103938 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.104526 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.104691 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:29.109549 (+ 4858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:53339: BlockingRecv error: recv got EOF from 127.9.231.65:53339 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 02:22:29 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:29 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:29.572540 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:29.562484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:36391)
0620 02:22:29.563124 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:29.563154 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:29.563351 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:29.563764 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:29.563768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:29.564176 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:29.564487 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:29.564499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.565603 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.565607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:29.566560 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:29.566572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.567364 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.567370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:29.567525 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:29.568273 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:29.568294 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:29.572255 (+ 3961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
W20260620 02:22:29.572724 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:29.562631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:36391 (local address 127.9.231.126:43409)
0620 02:22:29.562864 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:29.562871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:29.563236 (+ 365us) server_negotiation.cc:407] Connection header received
0620 02:22:29.563488 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:29.563492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:29.563556 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:29.563643 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:29.564711 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.565481 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.566716 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.567188 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.567304 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:29.572461 (+ 5157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:36391: BlockingRecv error: recv got EOF from 127.9.231.66:36391 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 02:22:29 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:29 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:29.990617 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:29.981581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:36059)
0620 02:22:29.982095 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:29.982112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:29.982294 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:29.982711 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:29.982714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:29.983184 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:29.983461 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:29.983472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.984582 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.984585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:29.985487 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:29.985498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.986249 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.986273 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:29.986394 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:29.987012 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:29.987028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:29.990390 (+ 3362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":289,"threads_started":1}
W20260620 02:22:29.990916 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:29.981668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:36059 (local address 127.9.231.126:43409)
0620 02:22:29.981878 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:29.981889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:29.982217 (+ 328us) server_negotiation.cc:407] Connection header received
0620 02:22:29.982438 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:29.982462 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:29.982548 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:29.982646 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:29.983603 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.984450 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.985626 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:29.986107 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:29.986224 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:29.990610 (+ 4386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:36059: BlockingRecv error: recv got EOF from 127.9.231.67:36059 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 02:22:30 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:30 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:30.120497 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:30.110855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:44917)
0620 02:22:30.111369 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:30.111385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:30.111544 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:30.111897 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:30.111902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:30.112349 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:30.112522 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:30.112533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.113553 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.113560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:30.114491 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:30.114501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.115329 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.115334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:30.115525 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:30.116180 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:30.116197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:30.120247 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":285,"threads_started":1}
W20260620 02:22:30.120668 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:30.110908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:44917 (local address 127.9.231.126:43409)
0620 02:22:30.111174 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:30.111182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:30.111476 (+ 294us) server_negotiation.cc:407] Connection header received
0620 02:22:30.111684 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:30.111688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:30.111759 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:30.111869 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:30.112672 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.113408 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.114631 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.115177 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.115316 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:30.120428 (+ 5112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:44917: BlockingRecv error: recv got EOF from 127.9.231.65:44917 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 02:22:30 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:30 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:30.583014 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:30.573543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:55463)
0620 02:22:30.574230 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:30.574249 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:30.574367 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:30.574715 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:30.574718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:30.575127 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:30.575306 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:30.575315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.576315 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.576319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:30.577287 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:30.577300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.578011 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.578016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:30.578185 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:30.578969 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:30.578989 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:30.582604 (+ 3615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":411,"threads_started":1}
W20260620 02:22:30.583312 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:30.573599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:55463 (local address 127.9.231.126:43409)
0620 02:22:30.573815 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:30.573821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:30.574358 (+ 537us) server_negotiation.cc:407] Connection header received
0620 02:22:30.574482 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:30.574486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:30.574554 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:30.574652 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:30.575459 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.576167 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.577426 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.577829 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.577924 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:30.582851 (+ 4927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:55463: BlockingRecv error: recv got EOF from 127.9.231.66:55463 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 02:22:30 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:30 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:31.002533 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:30.991830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:39643)
0620 02:22:30.992370 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:30.992386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:30.992549 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:30.993085 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:30.993092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:30.993649 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:30.993936 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:30.993950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.995408 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.995412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:30.996508 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:30.996522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.997503 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.997508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:30.997646 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:30.998325 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:30.998343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:31.002171 (+ 3828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":320,"threads_started":1}
W20260620 02:22:31.002717 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:30.991898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:39643 (local address 127.9.231.126:43409)
0620 02:22:30.992189 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:30.992197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:30.992520 (+ 323us) server_negotiation.cc:407] Connection header received
0620 02:22:30.992741 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:30.992745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:30.992865 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:30.992985 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:30.994089 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.995256 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.996667 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:30.997308 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:30.997407 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:31.002432 (+ 5025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:39643: BlockingRecv error: recv got EOF from 127.9.231.67:39643 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 02:22:31 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:31 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:31.131245 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:31.121600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:56221)
0620 02:22:31.122124 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:31.122140 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:31.122331 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:31.122680 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:31.122685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:31.123128 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:31.123291 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:31.123302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.124396 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.124401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:31.125391 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:31.125413 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.126133 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.126139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:31.126316 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:31.126943 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:31.126961 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:31.130801 (+ 3840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":294,"threads_started":1}
W20260620 02:22:31.131340 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:31.121710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:56221 (local address 127.9.231.126:43409)
0620 02:22:31.121983 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:31.121990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:31.122233 (+ 243us) server_negotiation.cc:407] Connection header received
0620 02:22:31.122460 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:31.122466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:31.122534 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:31.122636 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:31.123440 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.124205 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.125549 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.125972 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.126106 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:31.131017 (+ 4911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:56221: BlockingRecv error: recv got EOF from 127.9.231.65:56221 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 20 02:22:31 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:31 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:31.594370 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:31.584225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:40057)
0620 02:22:31.584811 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:31.584828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:31.584996 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:31.585360 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:31.585365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:31.585804 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:31.585959 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:31.585973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.587395 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.587402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:31.588365 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:31.588377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.589186 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.589192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:31.589397 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:31.590077 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:31.590096 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:31.594075 (+ 3979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":316,"threads_started":1}
W20260620 02:22:31.594707 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:31.584475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:40057 (local address 127.9.231.126:43409)
0620 02:22:31.584715 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:31.584722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:31.584915 (+ 193us) server_negotiation.cc:407] Connection header received
0620 02:22:31.585149 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:31.585159 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:31.585222 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:31.585327 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:31.586245 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.587238 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.588530 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:31.589011 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:31.589172 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:31.594357 (+ 5185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:40057: BlockingRecv error: recv got EOF from 127.9.231.66:40057 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 02:22:32 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:32 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.67@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:32.013551 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:32.003868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.67:40525)
0620 02:22:32.004390 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:32.004409 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:32.004521 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:32.005079 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:32.005082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:32.005474 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:32.005626 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:32.005634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.006787 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.006792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:32.007885 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:32.007899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.008661 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.008666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:32.008801 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.009427 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:32.009443 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:32.013204 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":269,"threads_started":1}
W20260620 02:22:32.013720 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:32.003968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:40525 (local address 127.9.231.126:43409)
0620 02:22:32.004136 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:32.004140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:32.004559 (+ 419us) server_negotiation.cc:407] Connection header received
0620 02:22:32.004831 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:32.004836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:32.004940 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:32.005058 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:32.005789 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.006603 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.008033 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.008482 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.008641 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.013495 (+ 4854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.67:40525: BlockingRecv error: recv got EOF from 127.9.231.67:40525 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 02:22:32 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:32 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.65@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:32.141868 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:32.132530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.65:37639)
0620 02:22:32.133058 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:32.133075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:32.133199 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:32.133495 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:32.133498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:32.133923 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:32.134106 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:32.134115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.135181 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.135185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:32.136108 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:32.136118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.136867 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.136872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:32.137020 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.137644 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:32.137670 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:32.141474 (+ 3804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":263,"threads_started":1}
W20260620 02:22:32.142064 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:32.132726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:37639 (local address 127.9.231.126:43409)
0620 02:22:32.133043 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:32.133056 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:32.133147 (+ 91us) server_negotiation.cc:407] Connection header received
0620 02:22:32.133308 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:32.133311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:32.133375 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:32.133455 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:32.134249 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.134981 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.136247 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.136687 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.136819 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.141758 (+ 4939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.65:37639: BlockingRecv error: recv got EOF from 127.9.231.65:37639 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Jun 20 02:22:32 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:22:32 dist-test-slave-9vg1 krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.231.66@KRBTEST.COM for oryx/127.9.231.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:22:32.606077 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:32.595859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:43409 (local address 127.9.231.66:46905)
0620 02:22:32.596393 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:32.596410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:32.596535 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:32.596939 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:32.596942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:32.597479 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:32.597806 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:32.597823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.599379 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.599384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:32.600511 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:32.600524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.601244 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.601249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:32.601386 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.602047 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:32.602064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:32.605818 (+ 3754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.231.126:43409: Server oryx/127.9.231.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":310,"threads_started":1}
W20260620 02:22:32.606325 19483 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:32.596102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46905 (local address 127.9.231.126:43409)
0620 02:22:32.596384 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:32.596394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:32.596502 (+ 108us) server_negotiation.cc:407] Connection header received
0620 02:22:32.596718 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:32.596722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:32.596791 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:32.596895 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:32.598067 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.599225 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.600705 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.601096 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.601200 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.606024 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.231.66:46905: BlockingRecv error: recv got EOF from 127.9.231.66:46905 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 02:22:32.634263 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19064
I20260620 02:22:32.647667 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:32.635535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.9.231.65:45209)
0620 02:22:32.636219 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:32.636226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:32.636255 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:22:32.636328 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:32.636335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:32.636573 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:32.636758 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:32.638218 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.639381 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.640936 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.641536 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.641655 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.644423 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:32.644432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:32.644446 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:32.644484 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:32.647150 (+ 2666us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.231.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:22:32.647462 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60002: Request ticket server kudu/127.9.231.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":352,"threads_started":1}
W20260620 02:22:32.647796 19528 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60002: Request ticket server kudu/127.9.231.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:22:32.647837 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19064: Not authorized: Client connection negotiation failed: client connection to 127.9.231.65:45209: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.231.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:22:32.648061 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 19064
I20260620 02:22:32.665995 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19216
I20260620 02:22:32.678781 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:32.667300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.9.231.66:35053)
0620 02:22:32.667971 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:32.667979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:32.668008 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:22:32.668072 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:32.668078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:32.668335 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:32.668551 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:32.669959 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.671020 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.672318 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.672891 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.673041 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.675616 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:32.675625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:32.675640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:32.675684 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:32.678065 (+ 2381us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.231.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:22:32.678447 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39398: Request ticket server kudu/127.9.231.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":377,"threads_started":1}
W20260620 02:22:32.678901 19529 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39398: Request ticket server kudu/127.9.231.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:22:32.678911 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19216: Not authorized: Client connection negotiation failed: client connection to 127.9.231.66:35053: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.231.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:22:32.679186 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 19216
I20260620 02:22:32.696540 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19349
I20260620 02:22:32.710325 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:32.697736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.9.231.67:46095)
0620 02:22:32.698379 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:32.698386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:32.698420 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:22:32.698492 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:32.698498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:32.698750 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:32.698914 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:32.700875 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.701975 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.703330 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:32.703958 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:32.704056 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:32.706784 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:32.706792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:32.706807 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:32.706848 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:32.709803 (+ 2955us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.231.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:22:32.710076 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35104: Request ticket server kudu/127.9.231.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":348,"threads_started":1}
W20260620 02:22:32.710453 19530 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35104: Request ticket server kudu/127.9.231.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:22:32.710454 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19349: Not authorized: Client connection negotiation failed: client connection to 127.9.231.67:46095: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.231.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:22:32.710592 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 19349
I20260620 02:22:32.727643 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 18994
I20260620 02:22:32.729074 19057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:32.849627 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 18994
2026-06-20T02:22:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21736 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T02:22:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:22:32Z Disabled control of system clock
I20260620 02:22:32.933331 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40793
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:22:33.278820 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:33.279206 19535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:33.279368 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:33.288090 19535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:22:33.288277 19535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:33.288335 19535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:33.288406 19535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:22:33.288457 19535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:22:33.298388 19535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:40793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:33.299892 19535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:33.301848 19535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:33.315758 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:33.315761 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:33.316041 19535 server_base.cc:1034] running on GCE node
W20260620 02:22:33.316430 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:33.317552 19535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:33.319149 19535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:33.320389 19535 hybrid_clock.cc:648] HybridClock initialized: now 1781922153320357 us; error 53 us; skew 500 ppm
I20260620 02:22:33.320894 19535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:33.328426 19535 webserver.cc:460] Webserver started at http://127.9.231.126:40271/ using document root <none> and password file <none>
I20260620 02:22:33.329254 19535 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:33.329401 19535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:33.329808 19535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:33.333276 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "71dadb33d1a24934b703f0ae23fc1954"
format_stamp: "Formatted at 2026-06-20 02:22:33 on dist-test-slave-9vg1"
server_key: "5f5ac4df5186e8a6264b326bc9bdca5c"
server_key_iv: "2c1d3580a8b0fba40b7c48271f0760d2"
server_key_version: "encryptionkey@0"
I20260620 02:22:33.334261 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "71dadb33d1a24934b703f0ae23fc1954"
format_stamp: "Formatted at 2026-06-20 02:22:33 on dist-test-slave-9vg1"
server_key: "5f5ac4df5186e8a6264b326bc9bdca5c"
server_key_iv: "2c1d3580a8b0fba40b7c48271f0760d2"
server_key_version: "encryptionkey@0"
I20260620 02:22:33.340387 19535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:22:33.344079 19549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:33.345685 19535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:22:33.345921 19535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "71dadb33d1a24934b703f0ae23fc1954"
format_stamp: "Formatted at 2026-06-20 02:22:33 on dist-test-slave-9vg1"
server_key: "5f5ac4df5186e8a6264b326bc9bdca5c"
server_key_iv: "2c1d3580a8b0fba40b7c48271f0760d2"
server_key_version: "encryptionkey@0"
I20260620 02:22:33.346181 19535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:33.385031 19535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:33.386353 19535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:33.386747 19535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:33.409780 19535 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:40793
I20260620 02:22:33.409825 19600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:40793 every 8 connection(s)
I20260620 02:22:33.411831 19535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:22:33.417392 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:33.418676 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 19535
I20260620 02:22:33.418919 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:22:33.419955 10141 external_mini_cluster.cc:1428] Setting key 7570eef57bacc28c0c611841e397e076
I20260620 02:22:33.433300 19601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954: Bootstrap starting.
I20260620 02:22:33.437319 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:33.422504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.9.231.126:40793)
0620 02:22:33.423811 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:33.423835 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:33.423887 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:22:33.425608 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:33.425638 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:33.426297 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:33.426734 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:33.428031 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:33.430129 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:33.431644 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:33.432690 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:33.432897 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:33.434278 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:33.434288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:33.434307 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:33.435049 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:33.435206 (+ 157us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:33.435243 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:33.435374 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:33.435645 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 02:22:33.436020 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":421,"threads_started":1}
I20260620 02:22:33.437539 19601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:33.438629 19601 log.cc:824] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:33.441756 19601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954: No bootstrap required, opened a new log
I20260620 02:22:33.450119 19601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71dadb33d1a24934b703f0ae23fc1954" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40793 } }
I20260620 02:22:33.450582 19601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:33.450686 19601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71dadb33d1a24934b703f0ae23fc1954, State: Initialized, Role: FOLLOWER
I20260620 02:22:33.451354 19601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71dadb33d1a24934b703f0ae23fc1954" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40793 } }
I20260620 02:22:33.451589 19601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:22:33.451756 19601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:22:33.451941 19601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:33.454277 19601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71dadb33d1a24934b703f0ae23fc1954" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40793 } }
I20260620 02:22:33.454907 19601 leader_election.cc:302] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71dadb33d1a24934b703f0ae23fc1954; no voters:
I20260620 02:22:33.455593 19601 leader_election.cc:288] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:22:33.455785 19606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:33.456920 19606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [term 1 LEADER]: Becoming Leader. State: Replica: 71dadb33d1a24934b703f0ae23fc1954, State: Running, Role: LEADER
I20260620 02:22:33.457571 19606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71dadb33d1a24934b703f0ae23fc1954" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40793 } }
I20260620 02:22:33.457932 19601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:22:33.460587 19607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71dadb33d1a24934b703f0ae23fc1954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71dadb33d1a24934b703f0ae23fc1954" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40793 } } }
I20260620 02:22:33.460973 19607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:33.461449 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71dadb33d1a24934b703f0ae23fc1954. Latest consensus state: current_term: 1 leader_uuid: "71dadb33d1a24934b703f0ae23fc1954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71dadb33d1a24934b703f0ae23fc1954" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40793 } } }
I20260620 02:22:33.464346 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:33.466089 19613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:22:33.470619 19613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:22:33.479758 19613 catalog_manager.cc:1347] Generated new cluster ID: 94c87c00b23341e0baafd9f00478c220
I20260620 02:22:33.479911 19613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:22:33.499480 19613 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:22:33.501654 19613 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:22:33.517073 19613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71dadb33d1a24934b703f0ae23fc1954: Generated new TSK 0
I20260620 02:22:33.518395 19613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:22:33.534945 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40793
--builtin_ntp_servers=127.9.231.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 02:22:33.879573 19625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:33.879892 19625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:33.880023 19625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:33.887498 19625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:33.887626 19625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:33.887773 19625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:22:33.897922 19625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:33.899456 19625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:33.901325 19625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:33.915616 19633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:33.915810 19631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:33.915827 19630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:33.916709 19625 server_base.cc:1034] running on GCE node
I20260620 02:22:33.917484 19625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:33.918529 19625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:33.919750 19625 hybrid_clock.cc:648] HybridClock initialized: now 1781922153919700 us; error 39 us; skew 500 ppm
I20260620 02:22:33.920117 19625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:33.922669 19625 webserver.cc:460] Webserver started at http://127.9.231.65:33517/ using document root <none> and password file <none>
I20260620 02:22:33.923520 19625 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:33.923619 19625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:33.923970 19625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:33.927030 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "15d75f55d71e49f7b6dfdae21d911179"
format_stamp: "Formatted at 2026-06-20 02:22:33 on dist-test-slave-9vg1"
server_key: "83706f0e7aab4cd2cb05b86c91f52546"
server_key_iv: "dc9d024c664a340f7386aa98e77d164a"
server_key_version: "encryptionkey@0"
I20260620 02:22:33.928009 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "15d75f55d71e49f7b6dfdae21d911179"
format_stamp: "Formatted at 2026-06-20 02:22:33 on dist-test-slave-9vg1"
server_key: "83706f0e7aab4cd2cb05b86c91f52546"
server_key_iv: "dc9d024c664a340f7386aa98e77d164a"
server_key_version: "encryptionkey@0"
I20260620 02:22:33.933482 19625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:22:33.937527 19639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:33.939253 19625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:22:33.939453 19625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "15d75f55d71e49f7b6dfdae21d911179"
format_stamp: "Formatted at 2026-06-20 02:22:33 on dist-test-slave-9vg1"
server_key: "83706f0e7aab4cd2cb05b86c91f52546"
server_key_iv: "dc9d024c664a340f7386aa98e77d164a"
server_key_version: "encryptionkey@0"
I20260620 02:22:33.939708 19625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:33.958830 19625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:33.959803 19625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:33.960114 19625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:33.961138 19625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:33.962730 19625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:33.962849 19625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:33.962980 19625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:33.963100 19625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:33.997442 19625 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:33859
I20260620 02:22:33.997486 19751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:33859 every 8 connection(s)
I20260620 02:22:33.998816 19625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:22:34.003098 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 19625
I20260620 02:22:34.003329 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:22:34.004050 10141 external_mini_cluster.cc:1428] Setting key a95a4524508166f8e12f9246bbdf0f6c
I20260620 02:22:34.006887 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40793
--builtin_ntp_servers=127.9.231.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:22:34.014199 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:34.001865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:53581 (local address 127.9.231.126:40793)
0620 02:22:34.002391 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:34.002399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:34.003832 (+ 1433us) server_negotiation.cc:407] Connection header received
0620 02:22:34.005742 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:34.005748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:34.005844 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:34.005970 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:34.007180 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.007915 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.010735 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.011328 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.011457 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:34.013355 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:34.013360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:34.013367 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:34.013391 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:34.013435 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:34.013448 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:34.013530 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:34.013812 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 02:22:34.013916 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
I20260620 02:22:34.015558 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:34.002419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40793 (local address 127.9.231.65:53581)
0620 02:22:34.003655 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:34.003701 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:34.005447 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:34.006216 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:34.006224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:34.006319 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:34.006909 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:34.006926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.008105 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.008113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:34.010459 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:34.010487 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.011593 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.011603 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:34.011968 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:34.012914 (+ 946us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:22:34.012919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:34.012981 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:22:34.013108 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:34.013557 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:34.013570 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:34.013582 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:22:34.013783 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 02:22:34.014265 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":349,"threads_started":1}
I20260620 02:22:34.017231 19752 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40793
I20260620 02:22:34.017563 19752 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:34.018523 19752 heartbeater.cc:511] Master 127.9.231.126:40793 requested a full tablet report, sending...
I20260620 02:22:34.020972 19566 ts_manager.cc:194] Registered new tserver with Master: 15d75f55d71e49f7b6dfdae21d911179 (127.9.231.65:33859)
I20260620 02:22:34.022475 19566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.65:53581
W20260620 02:22:34.346410 19756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:34.346704 19756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:34.346845 19756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:34.354444 19756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:34.354559 19756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:34.354732 19756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:22:34.365224 19756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:34.366794 19756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:34.368781 19756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:34.383800 19761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:34.383888 19762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:34.384658 19764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:34.384693 19756 server_base.cc:1034] running on GCE node
I20260620 02:22:34.385499 19756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:34.386688 19756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:34.387917 19756 hybrid_clock.cc:648] HybridClock initialized: now 1781922154387850 us; error 65 us; skew 500 ppm
I20260620 02:22:34.388275 19756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:34.390952 19756 webserver.cc:460] Webserver started at http://127.9.231.66:43917/ using document root <none> and password file <none>
I20260620 02:22:34.392088 19756 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:34.392267 19756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:34.392681 19756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:34.396348 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "ae97b7273d094c06ae812f583f16927b"
format_stamp: "Formatted at 2026-06-20 02:22:34 on dist-test-slave-9vg1"
server_key: "0bab0ee3f914733a44abcecc18cabd68"
server_key_iv: "a6c57563636bcd2198ceb6037cbf839d"
server_key_version: "encryptionkey@0"
I20260620 02:22:34.397167 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "ae97b7273d094c06ae812f583f16927b"
format_stamp: "Formatted at 2026-06-20 02:22:34 on dist-test-slave-9vg1"
server_key: "0bab0ee3f914733a44abcecc18cabd68"
server_key_iv: "a6c57563636bcd2198ceb6037cbf839d"
server_key_version: "encryptionkey@0"
I20260620 02:22:34.402597 19756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 02:22:34.406418 19770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:34.407989 19756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:22:34.408188 19756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "ae97b7273d094c06ae812f583f16927b"
format_stamp: "Formatted at 2026-06-20 02:22:34 on dist-test-slave-9vg1"
server_key: "0bab0ee3f914733a44abcecc18cabd68"
server_key_iv: "a6c57563636bcd2198ceb6037cbf839d"
server_key_version: "encryptionkey@0"
I20260620 02:22:34.408427 19756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:34.421125 19756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:34.421986 19756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:34.422309 19756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:34.423353 19756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:34.425024 19756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:34.425179 19756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:34.425338 19756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:34.425460 19756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:34.462108 19756 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:46327
I20260620 02:22:34.462152 19882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:46327 every 8 connection(s)
I20260620 02:22:34.463678 19756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:22:34.464691 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 19756
I20260620 02:22:34.464931 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:22:34.465525 10141 external_mini_cluster.cc:1428] Setting key 218124c9d33e59106e81e4e632e09742
I20260620 02:22:34.469103 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40793
--builtin_ntp_servers=127.9.231.84:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:22:34.477447 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:34.466729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:40489 (local address 127.9.231.126:40793)
0620 02:22:34.466966 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:34.466970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:34.468714 (+ 1744us) server_negotiation.cc:407] Connection header received
0620 02:22:34.470484 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:34.470489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:34.470573 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:34.470695 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:34.471827 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.472583 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.474349 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.474747 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.474889 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:34.476605 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:34.476609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:34.476613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:34.476632 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:34.476664 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:34.476675 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:34.476776 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:34.477015 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 02:22:34.477118 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 02:22:34.478502 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:34.467512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40793 (local address 127.9.231.66:40489)
0620 02:22:34.468569 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:34.468607 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:34.470212 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:34.470927 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:34.470936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:34.471028 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:34.471651 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:34.471668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.472736 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.472749 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:34.474182 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:34.474194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.474993 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.475000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:34.475367 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:34.476188 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:22:34.476193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:34.476261 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:22:34.476386 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:34.476774 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:34.476786 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:34.476795 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:22:34.476997 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 02:22:34.477405 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":352,"threads_started":1}
I20260620 02:22:34.480044 19883 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40793
I20260620 02:22:34.480370 19883 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:34.481145 19883 heartbeater.cc:511] Master 127.9.231.126:40793 requested a full tablet report, sending...
I20260620 02:22:34.482749 19566 ts_manager.cc:194] Registered new tserver with Master: ae97b7273d094c06ae812f583f16927b (127.9.231.66:46327)
I20260620 02:22:34.484030 19566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.66:40489
W20260620 02:22:34.805989 19886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:34.806283 19886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:34.806368 19886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:34.814092 19886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:34.814210 19886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:34.814318 19886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:22:34.823557 19886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:34.824851 19886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:34.826642 19886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:34.840193 19891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:34.840167 19892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:34.840497 19894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:34.841087 19886 server_base.cc:1034] running on GCE node
I20260620 02:22:34.841732 19886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:34.842787 19886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:34.844074 19886 hybrid_clock.cc:648] HybridClock initialized: now 1781922154843990 us; error 54 us; skew 500 ppm
I20260620 02:22:34.844514 19886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:34.847262 19886 webserver.cc:460] Webserver started at http://127.9.231.67:45401/ using document root <none> and password file <none>
I20260620 02:22:34.848045 19886 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:34.848196 19886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:34.848580 19886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:34.851979 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "02029c37fdbd4e00993b8b2595b7f627"
format_stamp: "Formatted at 2026-06-20 02:22:34 on dist-test-slave-9vg1"
server_key: "bc9ebd574feed0cef7272973287fee31"
server_key_iv: "76d351e18a3a499f699b199dece826db"
server_key_version: "encryptionkey@0"
I20260620 02:22:34.852913 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "02029c37fdbd4e00993b8b2595b7f627"
format_stamp: "Formatted at 2026-06-20 02:22:34 on dist-test-slave-9vg1"
server_key: "bc9ebd574feed0cef7272973287fee31"
server_key_iv: "76d351e18a3a499f699b199dece826db"
server_key_version: "encryptionkey@0"
I20260620 02:22:34.859519 19886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:22:34.864646 19900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:34.866696 19886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260620 02:22:34.866888 19886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "02029c37fdbd4e00993b8b2595b7f627"
format_stamp: "Formatted at 2026-06-20 02:22:34 on dist-test-slave-9vg1"
server_key: "bc9ebd574feed0cef7272973287fee31"
server_key_iv: "76d351e18a3a499f699b199dece826db"
server_key_version: "encryptionkey@0"
I20260620 02:22:34.867164 19886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:34.902772 19886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:34.903666 19886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:34.903936 19886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:34.904935 19886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:34.906518 19886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:34.906607 19886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:34.906690 19886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:34.906745 19886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:34.941255 19886 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:35727
I20260620 02:22:34.941354 20012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:35727 every 8 connection(s)
I20260620 02:22:34.942615 19886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:22:34.945716 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 19886
I20260620 02:22:34.945919 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:22:34.946399 10141 external_mini_cluster.cc:1428] Setting key 96b4977d65c4fae4dd0d03590255c41b
I20260620 02:22:34.956221 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:34.945544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:37677 (local address 127.9.231.126:40793)
0620 02:22:34.947032 (+ 1488us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:34.947041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:34.947422 (+ 381us) server_negotiation.cc:407] Connection header received
0620 02:22:34.949785 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:34.949789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:34.949860 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:34.949943 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:34.951226 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.951968 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.953571 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.953993 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.954090 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:34.955495 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:34.955500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:34.955503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:34.955523 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:34.955556 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:34.955567 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:34.955639 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:34.955891 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 02:22:34.955973 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1328}
I20260620 02:22:34.957270 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:34.946107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40793 (local address 127.9.231.67:37677)
0620 02:22:34.947261 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:34.947307 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:34.949439 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:34.950269 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:34.950279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:34.950374 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:34.951011 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:34.951031 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.952122 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.952126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:34.953431 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:34.953444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.954195 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.954200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:34.954429 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:34.955174 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:22:34.955176 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:34.955219 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:22:34.955308 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:34.955655 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:34.955662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:34.955668 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:22:34.955853 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 02:22:34.956169 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":347,"threads_started":1}
I20260620 02:22:34.958701 20013 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40793
I20260620 02:22:34.959116 20013 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:34.960006 20013 heartbeater.cc:511] Master 127.9.231.126:40793 requested a full tablet report, sending...
I20260620 02:22:34.961690 19566 ts_manager.cc:194] Registered new tserver with Master: 02029c37fdbd4e00993b8b2595b7f627 (127.9.231.67:35727)
I20260620 02:22:34.962306 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:22:34.962688 19566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.67:37677
W20260620 02:22:34.969532 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:34.967663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.9.231.126:40793)
0620 02:22:34.968074 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:34.968079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:34.968453 (+ 374us) server_negotiation.cc:407] Connection header received
0620 02:22:34.968654 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:34.968659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:34.968745 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:34.968846 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:34.969190 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38272: BlockingRecv error: recv got EOF from 127.0.0.1:38272 (error 108)
Metrics: {"server-negotiator.queue_time_us":290}
I20260620 02:22:34.970323 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19625
I20260620 02:22:34.981568 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:34.971352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.9.231.65:33859)
0620 02:22:34.974881 (+ 3529us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:34.974888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:34.974915 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:22:34.974971 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:34.974977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:34.975269 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:34.975461 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:34.975872 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.976869 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.978225 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:34.978776 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:34.978916 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:34.980096 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:34.980104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:34.980118 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:34.980706 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:34.980753 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:34.980772 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:34.980900 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:34.981051 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 02:22:34.981203 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3387,"thread_start_us":3284,"threads_started":1}
I20260620 02:22:34.982972 19747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:35.064903 19752 heartbeater.cc:503] Master 127.9.231.126:40793 was elected leader, sending a full tablet report...
I20260620 02:22:35.066720 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 19625
I20260620 02:22:35.084501 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19756
I20260620 02:22:35.092665 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:35.085429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.9.231.66:46327)
0620 02:22:35.085982 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:35.085990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:35.086018 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:22:35.086078 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:35.086085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:35.086316 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:35.086507 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:35.086928 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:35.088023 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:35.089233 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:35.089820 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:35.089966 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:35.091145 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:35.091158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:35.091177 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:35.091794 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:35.091841 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:35.091860 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:35.091997 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:35.092155 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 02:22:35.092318 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":292,"threads_started":1}
I20260620 02:22:35.094010 19878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:35.178977 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 19756
I20260620 02:22:35.196019 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19886
I20260620 02:22:35.204970 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:35.197013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.9.231.67:35727)
0620 02:22:35.197725 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:35.197732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:35.197761 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:22:35.197822 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:35.197829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:35.198040 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:35.198169 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:35.198646 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:35.199919 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:35.201172 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:35.201809 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:35.201922 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:35.203225 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:35.203242 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:35.203266 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:35.203949 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:35.204008 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:35.204028 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:35.204123 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:35.204345 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 02:22:35.204536 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":390,"threads_started":1}
I20260620 02:22:35.206391 20008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:35.293215 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 19886
I20260620 02:22:35.309521 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 19535
I20260620 02:22:35.310858 19596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:35.416355 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 19535
2026-06-20T02:22:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2559 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T02:22:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:22:35Z Disabled control of system clock
I20260620 02:22:35.492842 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38791
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:22:35.832777 20032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:35.833082 20032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:35.833165 20032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:35.840622 20032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:22:35.840747 20032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:35.840785 20032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:35.840834 20032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:22:35.840879 20032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:22:35.850333 20032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:35.851747 20032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:35.853590 20032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:35.866086 20038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:35.865993 20040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:35.866382 20037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:35.867138 20032 server_base.cc:1034] running on GCE node
I20260620 02:22:35.868075 20032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:35.869537 20032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:35.870805 20032 hybrid_clock.cc:648] HybridClock initialized: now 1781922155870756 us; error 41 us; skew 500 ppm
I20260620 02:22:35.871241 20032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:35.873569 20032 webserver.cc:460] Webserver started at http://127.9.231.126:44337/ using document root <none> and password file <none>
I20260620 02:22:35.874298 20032 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:35.874431 20032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:35.874768 20032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:35.877938 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "9bd778dc07844d52a7d76eab5c203ca5"
format_stamp: "Formatted at 2026-06-20 02:22:35 on dist-test-slave-9vg1"
server_key: "4294bc31d96095446d7724f3a4a51364"
server_key_iv: "87e615a1dca333e0e834603f3c5d6a06"
server_key_version: "encryptionkey@0"
I20260620 02:22:35.878777 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "9bd778dc07844d52a7d76eab5c203ca5"
format_stamp: "Formatted at 2026-06-20 02:22:35 on dist-test-slave-9vg1"
server_key: "4294bc31d96095446d7724f3a4a51364"
server_key_iv: "87e615a1dca333e0e834603f3c5d6a06"
server_key_version: "encryptionkey@0"
I20260620 02:22:35.884270 20032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:22:35.888170 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:35.889631 20032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:22:35.889881 20032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "9bd778dc07844d52a7d76eab5c203ca5"
format_stamp: "Formatted at 2026-06-20 02:22:35 on dist-test-slave-9vg1"
server_key: "4294bc31d96095446d7724f3a4a51364"
server_key_iv: "87e615a1dca333e0e834603f3c5d6a06"
server_key_version: "encryptionkey@0"
I20260620 02:22:35.890125 20032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:35.896170 20032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:35.897018 20032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:35.897341 20032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:35.916551 20032 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38791
I20260620 02:22:35.916577 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38791 every 8 connection(s)
I20260620 02:22:35.917996 20032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:22:35.921981 20098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:35.926813 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20032
I20260620 02:22:35.927021 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:22:35.927860 10141 external_mini_cluster.cc:1428] Setting key 68be961bf34abf6e475d0ed98e8f394e
I20260620 02:22:35.934584 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5: Bootstrap starting.
I20260620 02:22:35.937399 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:35.930176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.9.231.126:38791)
0620 02:22:35.931554 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:35.931570 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:35.931708 (+ 138us) server_negotiation.cc:407] Connection header received
0620 02:22:35.933508 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:35.933533 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:35.934029 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:35.934487 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:35.935682 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:35.935689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:35.935703 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:35.935725 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:35.935786 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:35.935802 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:35.936046 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:35.936197 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 02:22:35.936472 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":472,"threads_started":1}
I20260620 02:22:35.938416 20098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:35.939637 20098 log.cc:824] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:35.942495 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5: No bootstrap required, opened a new log
I20260620 02:22:35.950121 20098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd778dc07844d52a7d76eab5c203ca5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38791 } }
I20260620 02:22:35.950496 20098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:35.950637 20098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd778dc07844d52a7d76eab5c203ca5, State: Initialized, Role: FOLLOWER
I20260620 02:22:35.951303 20098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd778dc07844d52a7d76eab5c203ca5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38791 } }
I20260620 02:22:35.951536 20098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:22:35.951678 20098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:22:35.951848 20098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:35.954067 20098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd778dc07844d52a7d76eab5c203ca5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38791 } }
I20260620 02:22:35.954613 20098 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd778dc07844d52a7d76eab5c203ca5; no voters:
I20260620 02:22:35.955343 20098 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:22:35.955489 20103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:35.956259 20103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd778dc07844d52a7d76eab5c203ca5, State: Running, Role: LEADER
I20260620 02:22:35.956934 20103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd778dc07844d52a7d76eab5c203ca5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38791 } }
I20260620 02:22:35.957623 20098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:22:35.959757 20104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd778dc07844d52a7d76eab5c203ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd778dc07844d52a7d76eab5c203ca5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38791 } } }
I20260620 02:22:35.960186 20104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:35.961408 20105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd778dc07844d52a7d76eab5c203ca5. Latest consensus state: current_term: 1 leader_uuid: "9bd778dc07844d52a7d76eab5c203ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd778dc07844d52a7d76eab5c203ca5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38791 } } }
I20260620 02:22:35.961733 20105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:35.963030 20112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:22:35.967657 20112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:22:35.977167 20112 catalog_manager.cc:1347] Generated new cluster ID: f27e5e82560242ecb3bc85a8a11c8702
I20260620 02:22:35.977293 20112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:22:35.991349 20112 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:22:35.991896 20112 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:22:35.999526 20112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd778dc07844d52a7d76eab5c203ca5: Generated new TSK 0
I20260620 02:22:36.000612 20112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:22:36.016845 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38791
--builtin_ntp_servers=127.9.231.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:22:36.365945 20122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:36.366286 20122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:36.366456 20122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:36.373932 20122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:36.374101 20122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:36.374265 20122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:22:36.383785 20122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:36.385192 20122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:36.387144 20122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:36.401839 20128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:36.401849 20127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:36.402313 20130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:36.402861 20122 server_base.cc:1034] running on GCE node
I20260620 02:22:36.403613 20122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:36.404829 20122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:36.406113 20122 hybrid_clock.cc:648] HybridClock initialized: now 1781922156406074 us; error 70 us; skew 500 ppm
I20260620 02:22:36.406490 20122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:36.409235 20122 webserver.cc:460] Webserver started at http://127.9.231.65:34177/ using document root <none> and password file <none>
I20260620 02:22:36.410014 20122 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:36.410162 20122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:36.410492 20122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:36.413525 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "9190d60aa55540fc875d02eaa43454ed"
format_stamp: "Formatted at 2026-06-20 02:22:36 on dist-test-slave-9vg1"
server_key: "cd929c162662b9f09312106a42d99db1"
server_key_iv: "3d5ec65f77d8847be129046f9323a866"
server_key_version: "encryptionkey@0"
I20260620 02:22:36.414352 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "9190d60aa55540fc875d02eaa43454ed"
format_stamp: "Formatted at 2026-06-20 02:22:36 on dist-test-slave-9vg1"
server_key: "cd929c162662b9f09312106a42d99db1"
server_key_iv: "3d5ec65f77d8847be129046f9323a866"
server_key_version: "encryptionkey@0"
I20260620 02:22:36.419798 20122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 02:22:36.423750 20136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:36.425182 20122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:22:36.425379 20122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "9190d60aa55540fc875d02eaa43454ed"
format_stamp: "Formatted at 2026-06-20 02:22:36 on dist-test-slave-9vg1"
server_key: "cd929c162662b9f09312106a42d99db1"
server_key_iv: "3d5ec65f77d8847be129046f9323a866"
server_key_version: "encryptionkey@0"
I20260620 02:22:36.425670 20122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:36.431645 20122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:36.432476 20122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:36.432761 20122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:36.433744 20122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:36.435246 20122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:36.435369 20122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:36.435494 20122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:36.435580 20122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:36.470268 20122 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:33423
I20260620 02:22:36.470368 20248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:33423 every 8 connection(s)
I20260620 02:22:36.471863 20122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:22:36.474236 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20122
I20260620 02:22:36.474447 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:22:36.475310 10141 external_mini_cluster.cc:1428] Setting key e7b8b63c0c4893dab9383a4068f3b79b
I20260620 02:22:36.478111 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38791
--builtin_ntp_servers=127.9.231.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:22:36.480862 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:36.474694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:52313 (local address 127.9.231.126:38791)
0620 02:22:36.475244 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:36.475251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:36.476484 (+ 1233us) server_negotiation.cc:407] Connection header received
0620 02:22:36.478189 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:36.478197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:36.478265 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:36.478382 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:36.480017 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:36.480026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:36.480033 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:36.480056 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:36.480116 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:36.480133 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:36.480261 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:36.480427 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 02:22:36.480493 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
I20260620 02:22:36.481673 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:36.475249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38791 (local address 127.9.231.65:52313)
0620 02:22:36.476327 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:36.476378 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:36.477916 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:36.478733 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:36.478741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:36.478840 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:36.479645 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:22:36.479647 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:36.479745 (+ 98us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:22:36.479842 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:36.480234 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:36.480242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:36.480248 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:22:36.480460 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 02:22:36.480653 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":364,"threads_started":1}
I20260620 02:22:36.483260 20249 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38791
I20260620 02:22:36.483568 20249 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:36.484177 20249 heartbeater.cc:511] Master 127.9.231.126:38791 requested a full tablet report, sending...
I20260620 02:22:36.486644 20063 ts_manager.cc:194] Registered new tserver with Master: 9190d60aa55540fc875d02eaa43454ed (127.9.231.65:33423)
W20260620 02:22:36.811040 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:36.811375 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:36.811467 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:36.818673 20253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:36.818790 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:36.818903 20253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:22:36.828043 20253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:36.829473 20253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:36.831326 20253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:36.844815 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:36.844972 20261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:36.844858 20259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:36.846164 20253 server_base.cc:1034] running on GCE node
I20260620 02:22:36.846788 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:36.847885 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:36.849099 20253 hybrid_clock.cc:648] HybridClock initialized: now 1781922156849036 us; error 58 us; skew 500 ppm
I20260620 02:22:36.849396 20253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:36.851708 20253 webserver.cc:460] Webserver started at http://127.9.231.66:37095/ using document root <none> and password file <none>
I20260620 02:22:36.852386 20253 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:36.852475 20253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:36.852798 20253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:36.855777 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "7415b67ade9142c4b0e82125fb500882"
format_stamp: "Formatted at 2026-06-20 02:22:36 on dist-test-slave-9vg1"
server_key: "62846a2e842c74c7c2f25d29be7089c8"
server_key_iv: "888f08fba1444dec1b9dd8a5f6f07203"
server_key_version: "encryptionkey@0"
I20260620 02:22:36.856534 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "7415b67ade9142c4b0e82125fb500882"
format_stamp: "Formatted at 2026-06-20 02:22:36 on dist-test-slave-9vg1"
server_key: "62846a2e842c74c7c2f25d29be7089c8"
server_key_iv: "888f08fba1444dec1b9dd8a5f6f07203"
server_key_version: "encryptionkey@0"
I20260620 02:22:36.862718 20253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:22:36.867637 20267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:36.869330 20253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:22:36.869522 20253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "7415b67ade9142c4b0e82125fb500882"
format_stamp: "Formatted at 2026-06-20 02:22:36 on dist-test-slave-9vg1"
server_key: "62846a2e842c74c7c2f25d29be7089c8"
server_key_iv: "888f08fba1444dec1b9dd8a5f6f07203"
server_key_version: "encryptionkey@0"
I20260620 02:22:36.869781 20253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:36.875881 20253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:36.876775 20253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:36.877068 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:36.878135 20253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:36.879990 20253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:36.880087 20253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:36.880255 20253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:36.880322 20253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:36.916960 20253 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:38313
I20260620 02:22:36.916998 20379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:38313 every 8 connection(s)
I20260620 02:22:36.918318 20253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:22:36.924693 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20253
I20260620 02:22:36.924860 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:22:36.925374 10141 external_mini_cluster.cc:1428] Setting key 48ae4004ae065eede8d87703945aa3e2
I20260620 02:22:36.927660 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:36.921196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46653 (local address 127.9.231.126:38791)
0620 02:22:36.921384 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:36.921388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:36.923045 (+ 1657us) server_negotiation.cc:407] Connection header received
0620 02:22:36.924964 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:36.924969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:36.925028 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:36.925086 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:36.926865 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:36.926870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:36.926873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:36.926890 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:36.926921 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:36.926932 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:36.927055 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:36.927319 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 02:22:36.927386 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:22:36.928778 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38791
--builtin_ntp_servers=127.9.231.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:22:36.929705 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:36.921817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38791 (local address 127.9.231.66:46653)
0620 02:22:36.922874 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:36.922923 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:36.924605 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:36.925588 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:36.925596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:36.925700 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:36.926493 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:22:36.926496 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:36.926558 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:22:36.926660 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:36.927029 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:36.927040 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:36.927048 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:22:36.928035 (+ 987us) client_negotiation.cc:240] Negotiation successful
0620 02:22:36.928320 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":302,"threads_started":1}
I20260620 02:22:36.931412 20380 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38791
I20260620 02:22:36.931758 20380 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:36.932499 20380 heartbeater.cc:511] Master 127.9.231.126:38791 requested a full tablet report, sending...
I20260620 02:22:36.934221 20063 ts_manager.cc:194] Registered new tserver with Master: 7415b67ade9142c4b0e82125fb500882 (127.9.231.66:38313)
W20260620 02:22:37.271351 20383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:37.271651 20383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:37.271751 20383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:37.279369 20383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:37.279481 20383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:37.279698 20383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:22:37.289453 20383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:37.290913 20383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:37.292733 20383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:37.306497 20388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:37.306533 20391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:37.306556 20389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:37.308110 20383 server_base.cc:1034] running on GCE node
I20260620 02:22:37.308777 20383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:37.309804 20383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:37.311049 20383 hybrid_clock.cc:648] HybridClock initialized: now 1781922157311007 us; error 52 us; skew 500 ppm
I20260620 02:22:37.311452 20383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:22:37.313753 20383 webserver.cc:460] Webserver started at http://127.9.231.67:33821/ using document root <none> and password file <none>
I20260620 02:22:37.314459 20383 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:37.314592 20383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:37.314929 20383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:37.317898 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "61e4540c783d4e0fb80928bba62ccdbd"
format_stamp: "Formatted at 2026-06-20 02:22:37 on dist-test-slave-9vg1"
server_key: "6d7314e36019ab8053c90b87ebc6bd06"
server_key_iv: "265faf90181919034bbe4cccbe1ddf88"
server_key_version: "encryptionkey@0"
I20260620 02:22:37.318697 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "61e4540c783d4e0fb80928bba62ccdbd"
format_stamp: "Formatted at 2026-06-20 02:22:37 on dist-test-slave-9vg1"
server_key: "6d7314e36019ab8053c90b87ebc6bd06"
server_key_iv: "265faf90181919034bbe4cccbe1ddf88"
server_key_version: "encryptionkey@0"
I20260620 02:22:37.323827 20383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 02:22:37.327245 20397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:37.328636 20383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:22:37.328806 20383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "61e4540c783d4e0fb80928bba62ccdbd"
format_stamp: "Formatted at 2026-06-20 02:22:37 on dist-test-slave-9vg1"
server_key: "6d7314e36019ab8053c90b87ebc6bd06"
server_key_iv: "265faf90181919034bbe4cccbe1ddf88"
server_key_version: "encryptionkey@0"
I20260620 02:22:37.329023 20383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:37.334897 20383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:37.335736 20383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:37.336014 20383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:37.336925 20383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:37.338384 20383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:37.338507 20383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:37.338650 20383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:37.338738 20383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:37.372951 20383 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:46053
I20260620 02:22:37.372993 20509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:46053 every 8 connection(s)
I20260620 02:22:37.374454 20383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:22:37.374745 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20383
I20260620 02:22:37.374904 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:22:37.375602 10141 external_mini_cluster.cc:1428] Setting key 47593ec94a3381aa79e321adc1ec972c
I20260620 02:22:37.383446 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:37.377393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:43541 (local address 127.9.231.126:38791)
0620 02:22:37.378556 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:37.378564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:37.379378 (+ 814us) server_negotiation.cc:407] Connection header received
0620 02:22:37.381186 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:37.381190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:37.381248 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:37.381360 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:37.382723 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:37.382727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:37.382731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:37.382748 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:37.382793 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:37.382805 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:37.382915 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:37.383140 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 02:22:37.383208 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007}
I20260620 02:22:37.384397 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:37.378087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38791 (local address 127.9.231.67:43541)
0620 02:22:37.379186 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:37.379240 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:37.380901 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:37.381549 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:37.381557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:37.381651 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:37.382376 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:22:37.382379 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:37.382417 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:22:37.382508 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:37.382898 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:37.382906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:37.382911 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:22:37.383184 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 02:22:37.383387 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":408,"threads_started":1}
I20260620 02:22:37.385864 20510 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38791
I20260620 02:22:37.386206 20510 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:37.386871 20510 heartbeater.cc:511] Master 127.9.231.126:38791 requested a full tablet report, sending...
I20260620 02:22:37.388435 20063 ts_manager.cc:194] Registered new tserver with Master: 61e4540c783d4e0fb80928bba62ccdbd (127.9.231.67:46053)
I20260620 02:22:37.391211 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:22:37.398005 20251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:22:37.396322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.9.231.126:38791)
0620 02:22:37.396494 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:37.396521 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:37.397070 (+ 549us) server_negotiation.cc:407] Connection header received
0620 02:22:37.397209 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:37.397212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:37.397299 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:37.397393 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:37.397712 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58338: BlockingRecv error: recv got EOF from 127.0.0.1:58338 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:22:37.398830 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20122
I20260620 02:22:37.402304 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:37.399797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.9.231.65:33423)
0620 02:22:37.400321 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:37.400329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:37.400356 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:22:37.400415 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:37.400421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:37.400631 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:37.400762 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:37.401636 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:37.401643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:37.401654 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:37.401672 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:37.401715 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:37.401731 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:37.401881 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:37.401962 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 02:22:37.402037 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":254,"threads_started":1}
I20260620 02:22:37.403898 20244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:37.487200 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20122
I20260620 02:22:37.504312 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20253
I20260620 02:22:37.508183 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:37.505371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.9.231.66:38313)
0620 02:22:37.505909 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:37.505916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:37.505943 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:22:37.506002 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:37.506008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:37.506219 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:37.506356 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:37.507360 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:37.507367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:37.507379 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:37.507399 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:37.507444 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:37.507458 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:37.507572 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:37.507754 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 02:22:37.507877 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
I20260620 02:22:37.509502 20375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:37.591897 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20253
I20260620 02:22:37.608875 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20383
I20260620 02:22:37.613040 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:37.609983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.9.231.67:46053)
0620 02:22:37.610653 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:37.610659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:37.610692 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:22:37.610770 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:37.610777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:37.610988 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:37.611203 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:37.612255 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:37.612262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:37.612273 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:22:37.612284 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:37.612321 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:22:37.612335 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:37.612467 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:37.612577 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 02:22:37.612702 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":371,"threads_started":1}
I20260620 02:22:37.614290 20505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:37.696353 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20383
I20260620 02:22:37.714112 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20032
I20260620 02:22:37.715492 20093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:37.820086 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20032
2026-06-20T02:22:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2402 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:22:37 dist-test-slave-9vg1 krb5kdc[20528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:22:37 dist-test-slave-9vg1 krb5kdc[20528](info): set up 2 sockets
Jun 20 02:22:37 dist-test-slave-9vg1 krb5kdc[20528](info): commencing operation
krb5kdc: starting...
W20260620 02:22:38.941228 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:22:39 dist-test-slave-9vg1 krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:22:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:22:39.188918 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33713
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:39.525098 20541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:39.525453 20541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:39.525624 20541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:39.532981 20541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:22:39.533139 20541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:39.533259 20541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:39.533362 20541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:22:39.533449 20541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:22:39.543665 20541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:39.545091 20541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:39.546406 20541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:39.558535 20549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:39.558576 20546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:39.558636 20547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:39.559497 20541 server_base.cc:1034] running on GCE node
I20260620 02:22:39.560341 20541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:39.561916 20541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:39.563175 20541 hybrid_clock.cc:648] HybridClock initialized: now 1781922159563110 us; error 52 us; skew 500 ppm
I20260620 02:22:39.563544 20541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:39 dist-test-slave-9vg1 krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922159, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:39.571789 20541 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:22:39.573428 20541 webserver.cc:460] Webserver started at http://127.9.231.126:34527/ using document root <none> and password file <none>
I20260620 02:22:39.574126 20541 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:39.574227 20541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:39.574477 20541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:39.577538 20541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "88cc650c3328436ab2542d38dd11c8f8"
format_stamp: "Formatted at 2026-06-20 02:22:39 on dist-test-slave-9vg1"
server_key: "b38862367a2a4dc0abdfc4e65dc9d093"
server_key_iv: "ebcf27b984fec43e766e957709855552"
server_key_version: "encryptionkey@0"
I20260620 02:22:39.578258 20541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "88cc650c3328436ab2542d38dd11c8f8"
format_stamp: "Formatted at 2026-06-20 02:22:39 on dist-test-slave-9vg1"
server_key: "b38862367a2a4dc0abdfc4e65dc9d093"
server_key_iv: "ebcf27b984fec43e766e957709855552"
server_key_version: "encryptionkey@0"
I20260620 02:22:39.583565 20541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:22:39.587337 20556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:39.588701 20541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 02:22:39.588871 20541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "88cc650c3328436ab2542d38dd11c8f8"
format_stamp: "Formatted at 2026-06-20 02:22:39 on dist-test-slave-9vg1"
server_key: "b38862367a2a4dc0abdfc4e65dc9d093"
server_key_iv: "ebcf27b984fec43e766e957709855552"
server_key_version: "encryptionkey@0"
I20260620 02:22:39.589046 20541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:39.622217 20541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:39.626627 20541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:39.626890 20541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:39.646415 20541 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33713
I20260620 02:22:39.646459 20607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33713 every 8 connection(s)
I20260620 02:22:39.647919 20541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:22:39.651964 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:39.653512 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20541
I20260620 02:22:39.653744 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:22:39.654574 10141 external_mini_cluster.cc:1428] Setting key 99a2481c500067ea81f5eecc77e3fab9
I20260620 02:22:39.667035 20608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8: Bootstrap starting.
Jun 20 02:22:39 dist-test-slave-9vg1 krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:39.670728 20608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:39.672078 20608 log.cc:824] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:39.675213 20608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8: No bootstrap required, opened a new log
I20260620 02:22:39.678056 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:39.656693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.9.231.126:33713)
0620 02:22:39.657532 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:39.657543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:39.657585 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:22:39.659241 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:39.659264 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:39.659667 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:39.660007 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:39.661888 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:39.663252 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:39.665052 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:39.665714 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:39.665860 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:39.670471 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:39.670479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:39.670495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:39.670535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:39.673817 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:39.674564 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:39.674570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:39.674577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:39.674747 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:39.675171 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:39.675177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:39.675180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:39.675807 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:39.676136 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:39.676560 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 02:22:39.677024 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":380,"threads_started":1}
I20260620 02:22:39.683405 20608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cc650c3328436ab2542d38dd11c8f8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33713 } }
I20260620 02:22:39.683782 20608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:39.683889 20608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88cc650c3328436ab2542d38dd11c8f8, State: Initialized, Role: FOLLOWER
I20260620 02:22:39.684525 20608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cc650c3328436ab2542d38dd11c8f8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33713 } }
I20260620 02:22:39.684831 20608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:22:39.684990 20608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:22:39.685206 20608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:39.687630 20608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cc650c3328436ab2542d38dd11c8f8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33713 } }
I20260620 02:22:39.688160 20608 leader_election.cc:302] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88cc650c3328436ab2542d38dd11c8f8; no voters:
I20260620 02:22:39.688807 20608 leader_election.cc:288] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:22:39.688949 20613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:39.689613 20613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [term 1 LEADER]: Becoming Leader. State: Replica: 88cc650c3328436ab2542d38dd11c8f8, State: Running, Role: LEADER
I20260620 02:22:39.690181 20613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cc650c3328436ab2542d38dd11c8f8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33713 } }
I20260620 02:22:39.690927 20608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:22:39.693099 20614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88cc650c3328436ab2542d38dd11c8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cc650c3328436ab2542d38dd11c8f8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33713 } } }
I20260620 02:22:39.693555 20614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:39.693533 20615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88cc650c3328436ab2542d38dd11c8f8. Latest consensus state: current_term: 1 leader_uuid: "88cc650c3328436ab2542d38dd11c8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cc650c3328436ab2542d38dd11c8f8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33713 } } }
I20260620 02:22:39.693902 20615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:22:39.698055 20621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:22:39.703477 20621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:22:39.712673 20621 catalog_manager.cc:1347] Generated new cluster ID: 80dd52ae04ad492b879b6c733b3fa2a7
I20260620 02:22:39.712812 20621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:22:39.785924 20621 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:22:39.787500 20621 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:22:39.800168 20621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8: Generated new TSK 0
I20260620 02:22:39.801101 20621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:22:39.855742 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33713
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:40.199378 20634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:40.199704 20634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:40.199807 20634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:40.207520 20634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:40.207662 20634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:40.207787 20634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:22:40.218271 20634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:40.219650 20634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:40.220947 20634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:40.234699 20640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:40.234814 20639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:40.234786 20642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:40.235821 20634 server_base.cc:1034] running on GCE node
I20260620 02:22:40.236564 20634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:40.237622 20634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:40.238866 20634 hybrid_clock.cc:648] HybridClock initialized: now 1781922160238822 us; error 33 us; skew 500 ppm
I20260620 02:22:40.239292 20634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:40 dist-test-slave-9vg1 krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:40.242429 20634 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:22:40.244066 20634 webserver.cc:460] Webserver started at http://127.9.231.65:39079/ using document root <none> and password file <none>
I20260620 02:22:40.244765 20634 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:40.244863 20634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:40.245111 20634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:40.248588 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "107df6b298774e178197728cadb9b515"
format_stamp: "Formatted at 2026-06-20 02:22:40 on dist-test-slave-9vg1"
server_key: "fa4372051865efc4afcd91e970e5527e"
server_key_iv: "ccee6ec2b65303bcdbbe2a8d4638e204"
server_key_version: "encryptionkey@0"
I20260620 02:22:40.249336 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "107df6b298774e178197728cadb9b515"
format_stamp: "Formatted at 2026-06-20 02:22:40 on dist-test-slave-9vg1"
server_key: "fa4372051865efc4afcd91e970e5527e"
server_key_iv: "ccee6ec2b65303bcdbbe2a8d4638e204"
server_key_version: "encryptionkey@0"
I20260620 02:22:40.254788 20634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:22:40.258435 20649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:40.259940 20634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:22:40.260088 20634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "107df6b298774e178197728cadb9b515"
format_stamp: "Formatted at 2026-06-20 02:22:40 on dist-test-slave-9vg1"
server_key: "fa4372051865efc4afcd91e970e5527e"
server_key_iv: "ccee6ec2b65303bcdbbe2a8d4638e204"
server_key_version: "encryptionkey@0"
I20260620 02:22:40.260275 20634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:40.273263 20634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:40.277501 20634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:40.277776 20634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:40.278685 20634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:40.280141 20634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:40.280220 20634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:40.280304 20634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:40.280373 20634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:40.315659 20634 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:40499
I20260620 02:22:40.315712 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:40499 every 8 connection(s)
I20260620 02:22:40.316952 20634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:22:40.320016 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20634
I20260620 02:22:40.320235 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:22:40.320860 10141 external_mini_cluster.cc:1428] Setting key d069582f324fc5ee85e7bbc35acf7854
Jun 20 02:22:40 dist-test-slave-9vg1 krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:40.339156 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:40.319610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:56147 (local address 127.9.231.126:33713)
0620 02:22:40.320099 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:40.320107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:40.321163 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 02:22:40.323236 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:40.323242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:40.323321 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:40.323442 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:40.324884 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.325988 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.327564 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.328280 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.328420 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:40.332166 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:40.332176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:40.332184 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:40.332229 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:40.335787 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:40.336530 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:40.336539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:40.336545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:40.336703 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:40.337111 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:40.337119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:40.337125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:40.337518 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:40.337677 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:40.338376 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 02:22:40.338627 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260620 02:22:40.339939 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:40.320167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33713 (local address 127.9.231.65:56147)
0620 02:22:40.320993 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:40.321036 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:40.322914 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:40.323622 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:40.323629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:40.324158 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:40.324715 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:40.324732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.326136 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.326139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:40.327420 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:40.327432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.328476 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.328481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:40.328677 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:40.329360 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:40.329383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:40.331942 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:40.335931 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:40.335938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:40.335953 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:40.336384 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:40.336809 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:40.336812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:40.336814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:40.336992 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:40.337632 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:40.337638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:40.337986 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 02:22:40.338208 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 02:22:40.338614 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":363,"threads_started":1}
I20260620 02:22:40.341502 20762 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33713
I20260620 02:22:40.341879 20762 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:40.342852 20762 heartbeater.cc:511] Master 127.9.231.126:33713 requested a full tablet report, sending...
I20260620 02:22:40.345127 20573 ts_manager.cc:194] Registered new tserver with Master: 107df6b298774e178197728cadb9b515 (127.9.231.65:40499)
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:22:40.346624 20573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:56147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:22:40.378073 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33713
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:40.721570 20768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:40.721884 20768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:40.722079 20768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:40.729518 20768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:40.729673 20768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:40.729889 20768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:22:40.739807 20768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:40.741178 20768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:40.742501 20768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:40.756094 20774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:40.756140 20776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:40.756453 20773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:40.757217 20768 server_base.cc:1034] running on GCE node
I20260620 02:22:40.758003 20768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:40.759023 20768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:40.760298 20768 hybrid_clock.cc:648] HybridClock initialized: now 1781922160760231 us; error 60 us; skew 500 ppm
I20260620 02:22:40.760615 20768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:40 dist-test-slave-9vg1 krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:40.763842 20768 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:22:40.765440 20768 webserver.cc:460] Webserver started at http://127.9.231.66:42897/ using document root <none> and password file <none>
I20260620 02:22:40.766147 20768 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:40.766300 20768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:40.766609 20768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:40.769830 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "d3932ff7e9a84d9c88258b288b851a23"
format_stamp: "Formatted at 2026-06-20 02:22:40 on dist-test-slave-9vg1"
server_key: "07976b813872fad91beed548bfd71731"
server_key_iv: "958a5d6dac25450de71c22274f0db2d2"
server_key_version: "encryptionkey@0"
I20260620 02:22:40.770641 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "d3932ff7e9a84d9c88258b288b851a23"
format_stamp: "Formatted at 2026-06-20 02:22:40 on dist-test-slave-9vg1"
server_key: "07976b813872fad91beed548bfd71731"
server_key_iv: "958a5d6dac25450de71c22274f0db2d2"
server_key_version: "encryptionkey@0"
I20260620 02:22:40.776168 20768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:22:40.779830 20783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:40.781273 20768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:22:40.781469 20768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "d3932ff7e9a84d9c88258b288b851a23"
format_stamp: "Formatted at 2026-06-20 02:22:40 on dist-test-slave-9vg1"
server_key: "07976b813872fad91beed548bfd71731"
server_key_iv: "958a5d6dac25450de71c22274f0db2d2"
server_key_version: "encryptionkey@0"
I20260620 02:22:40.781692 20768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:40.805327 20768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:40.809396 20768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:40.809741 20768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:40.810737 20768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:40.812182 20768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:40.812288 20768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:40.812445 20768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:40.812520 20768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:40.846485 20768 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:34409
I20260620 02:22:40.846530 20895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:34409 every 8 connection(s)
I20260620 02:22:40.847922 20768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:22:40.852208 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20768
I20260620 02:22:40.852444 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:22:40.852995 10141 external_mini_cluster.cc:1428] Setting key 2dbd41ab1258d0f331c4ff6295fd3d1b
Jun 20 02:22:40 dist-test-slave-9vg1 krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:40.873637 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:40.850814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:58951 (local address 127.9.231.126:33713)
0620 02:22:40.851394 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:40.851399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:40.852524 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 02:22:40.854821 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:40.854827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:40.854898 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:40.855017 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:40.857260 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.858095 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.860310 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.861014 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.861149 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:40.865957 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:40.865967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:40.865973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:40.866026 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:40.869728 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:40.870668 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:40.870677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:40.870683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:40.870855 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:40.871450 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:40.871459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:40.871465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:40.871844 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:40.871997 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:40.872770 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 02:22:40.873022 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":335,"threads_started":1}
I20260620 02:22:40.874303 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:40.851424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33713 (local address 127.9.231.66:58951)
0620 02:22:40.852311 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:40.852367 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:40.854538 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:40.855325 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:40.855333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:40.856171 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:40.857058 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:40.857088 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.858296 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.858300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:40.860111 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:40.860130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:40.861442 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:40.861450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:40.861755 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:40.862595 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:40.862629 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:40.865640 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:40.869947 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:40.869959 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:40.869974 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:40.870502 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:40.871008 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:40.871012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:40.871015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:40.871323 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:40.871985 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:40.871992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:40.872365 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 02:22:40.872605 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 02:22:40.873033 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":342,"threads_started":1}
I20260620 02:22:40.875840 20896 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33713
I20260620 02:22:40.876178 20896 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:40.876909 20896 heartbeater.cc:511] Master 127.9.231.126:33713 requested a full tablet report, sending...
I20260620 02:22:40.878366 20573 ts_manager.cc:194] Registered new tserver with Master: d3932ff7e9a84d9c88258b288b851a23 (127.9.231.66:34409)
I20260620 02:22:40.879282 20573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:58951
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:22:40.912187 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33713
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:22:41.243821 20902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:22:41.244189 20902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:22:41.244374 20902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:22:41.251783 20902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:22:41.251969 20902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:22:41.252173 20902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:22:41.262179 20902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:22:41.263688 20902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:22:41.265065 20902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:22:41.278663 20910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:41.278739 20908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:22:41.278775 20907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:22:41.279364 20902 server_base.cc:1034] running on GCE node
I20260620 02:22:41.280102 20902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:22:41.281168 20902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:22:41.282370 20902 hybrid_clock.cc:648] HybridClock initialized: now 1781922161282318 us; error 37 us; skew 500 ppm
I20260620 02:22:41.282696 20902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:22:41 dist-test-slave-9vg1 krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922161, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:22:41.286005 20902 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:22:41.287614 20902 webserver.cc:460] Webserver started at http://127.9.231.67:46605/ using document root <none> and password file <none>
I20260620 02:22:41.288300 20902 fs_manager.cc:359] Metadata directory not provided
I20260620 02:22:41.288396 20902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:22:41.288640 20902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:22:41.291708 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "c4c25b523e644312ae31d1e6ae8a097f"
format_stamp: "Formatted at 2026-06-20 02:22:41 on dist-test-slave-9vg1"
server_key: "15c0aad9f77f1fc774eb0bb167178780"
server_key_iv: "f93c1c2e66a683941351e6874918a40d"
server_key_version: "encryptionkey@0"
I20260620 02:22:41.292474 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "c4c25b523e644312ae31d1e6ae8a097f"
format_stamp: "Formatted at 2026-06-20 02:22:41 on dist-test-slave-9vg1"
server_key: "15c0aad9f77f1fc774eb0bb167178780"
server_key_iv: "f93c1c2e66a683941351e6874918a40d"
server_key_version: "encryptionkey@0"
I20260620 02:22:41.298126 20902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:22:41.301765 20917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:41.303328 20902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:22:41.303507 20902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "c4c25b523e644312ae31d1e6ae8a097f"
format_stamp: "Formatted at 2026-06-20 02:22:41 on dist-test-slave-9vg1"
server_key: "15c0aad9f77f1fc774eb0bb167178780"
server_key_iv: "f93c1c2e66a683941351e6874918a40d"
server_key_version: "encryptionkey@0"
I20260620 02:22:41.303719 20902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:22:41.337473 20902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:22:41.341740 20902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:22:41.342042 20902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:22:41.343050 20902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:22:41.344447 20902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:22:41.344547 20902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:41.344712 20902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:22:41.344784 20902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:22:41.350083 20762 heartbeater.cc:503] Master 127.9.231.126:33713 was elected leader, sending a full tablet report...
I20260620 02:22:41.379151 20902 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:39907
I20260620 02:22:41.379208 21029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:39907 every 8 connection(s)
I20260620 02:22:41.380496 20902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:22:41.387497 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 20902
I20260620 02:22:41.387704 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:22:41.388209 10141 external_mini_cluster.cc:1428] Setting key 3fea80f3dd5535ed5ec1219b4d3dadaa
Jun 20 02:22:41 dist-test-slave-9vg1 krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922161, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:22:41.402806 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.383193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:36157 (local address 127.9.231.126:33713)
0620 02:22:41.383633 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.383638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.384585 (+ 947us) server_negotiation.cc:407] Connection header received
0620 02:22:41.386582 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.386588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.386700 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.386844 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:41.388721 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.389846 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.392015 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.392443 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.392579 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.396457 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:41.396462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:41.396466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:41.396527 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:41.399474 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.400313 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.400325 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.400328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.400448 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.400912 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.400917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.400922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.401225 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:41.401394 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.402153 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.402391 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":208,"threads_started":1}
I20260620 02:22:41.403980 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.383754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33713 (local address 127.9.231.67:36157)
0620 02:22:41.384424 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:41.384466 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:41.386252 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:41.387179 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:41.387188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:41.387860 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:22:41.388523 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.388544 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.390011 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.390015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.391798 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.391881 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.392697 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.392706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.392967 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.393690 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:22:41.393726 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:22:41.396205 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:22:41.399642 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:41.399653 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:41.399674 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:41.400166 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:41.400570 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:22:41.400575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:22:41.400579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:22:41.400785 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:22:41.401359 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:22:41.401367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:22:41.401752 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 02:22:41.402032 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 02:22:41.402621 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":267,"threads_started":1}
I20260620 02:22:41.405488 21030 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33713
I20260620 02:22:41.405843 21030 heartbeater.cc:464] Registering TS with master...
I20260620 02:22:41.406641 21030 heartbeater.cc:511] Master 127.9.231.126:33713 requested a full tablet report, sending...
I20260620 02:22:41.408356 20573 ts_manager.cc:194] Registered new tserver with Master: c4c25b523e644312ae31d1e6ae8a097f (127.9.231.67:39907)
I20260620 02:22:41.409313 20573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:36157
I20260620 02:22:41.417128 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:22:41.436328 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.422854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.9.231.126:33713)
0620 02:22:41.423029 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.423034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.423598 (+ 564us) server_negotiation.cc:407] Connection header received
0620 02:22:41.423754 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.423758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.423841 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.423936 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:41.425855 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.426576 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.427928 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.428331 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.428440 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.430777 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:41.430782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:41.430785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:41.430822 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:41.433715 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.434361 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.434366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.434368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.434498 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.434986 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.434990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.434992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.435398 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:41.435591 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.435914 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.436089 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:22:41.440641 20573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:22:41.447767 20573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:22:41.492637 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.460811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.9.231.65:40499)
0620 02:22:41.482340 (+ 21529us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.482353 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.482392 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:22:41.482517 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.482528 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.482874 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.483154 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:41.484173 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.486549 (+ 2376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.489533 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.491277 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.491409 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.491641 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.492093 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.492288 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21286,"thread_start_us":22600,"threads_started":1}
I20260620 02:22:41.492610 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.460158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40499 (local address 127.0.0.1:38770)
0620 02:22:41.473342 (+ 13184us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:41.473406 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:41.473703 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:41.483210 (+ 9507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:41.483221 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:41.483266 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:41.483860 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.483877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.486846 (+ 2969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.486855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.489353 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.489367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.491532 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.491539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.491755 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.491810 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 02:22:41.491970 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 02:22:41.492164 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12783,"thread_start_us":12551,"threads_started":1}
I20260620 02:22:41.494922 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.460162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:39907 (local address 127.0.0.1:59564)
0620 02:22:41.486738 (+ 26576us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:41.486760 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:41.486923 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:41.487806 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:41.487811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:41.487839 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:41.488308 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.488321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.490034 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.490039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.492288 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.492302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.494067 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.494073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.494252 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.494294 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:22:41.494383 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:22:41.494513 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26434,"mutex_wait_us":34,"thread_start_us":26758,"threads_started":1}
I20260620 02:22:41.495585 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.460697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:34409 (local address 127.0.0.1:46446)
0620 02:22:41.487260 (+ 26563us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:41.487277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:41.487390 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:41.488174 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:41.488180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:41.488209 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:41.488771 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.488784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.490574 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.490579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.492972 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.492990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.494733 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.494739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.494926 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.494957 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:22:41.495047 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:22:41.495206 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26443,"mutex_wait_us":70,"thread_start_us":25718,"threads_started":1}
I20260620 02:22:41.495707 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.460638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.9.231.67:39907)
0620 02:22:41.484157 (+ 23519us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.484169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.487133 (+ 2964us) server_negotiation.cc:407] Connection header received
0620 02:22:41.487254 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.487262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.487568 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.487818 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:41.488520 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.489871 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.492478 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.493854 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.494011 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.495107 (+ 1096us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.495186 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.495361 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23278,"thread_start_us":26577,"threads_started":1}
I20260620 02:22:41.496337 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.461268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.9.231.66:34409)
0620 02:22:41.484623 (+ 23355us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.484635 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.487557 (+ 2922us) server_negotiation.cc:407] Connection header received
0620 02:22:41.487666 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.487676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.487913 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.488100 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:41.488970 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.490415 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.493196 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.494493 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.494652 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.494852 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.495828 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.496007 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23197,"thread_start_us":23088,"threads_started":1}
I20260620 02:22:41.497366 20697 tablet_service.cc:1467] Processing CreateTablet for tablet da15cb0b64944a59aec608722db2b066 (DEFAULT_TABLE table=test-table [id=3ecb6d104fda485ea597e6ec30604c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:41.498714 20697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da15cb0b64944a59aec608722db2b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:41.498927 20965 tablet_service.cc:1467] Processing CreateTablet for tablet da15cb0b64944a59aec608722db2b066 (DEFAULT_TABLE table=test-table [id=3ecb6d104fda485ea597e6ec30604c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:41.500341 20965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da15cb0b64944a59aec608722db2b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:41.502784 20831 tablet_service.cc:1467] Processing CreateTablet for tablet da15cb0b64944a59aec608722db2b066 (DEFAULT_TABLE table=test-table [id=3ecb6d104fda485ea597e6ec30604c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:22:41.504586 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da15cb0b64944a59aec608722db2b066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:22:41.511713 21049 tablet_bootstrap.cc:492] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Bootstrap starting.
I20260620 02:22:41.515450 21049 tablet_bootstrap.cc:654] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:41.515846 21050 tablet_bootstrap.cc:492] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Bootstrap starting.
I20260620 02:22:41.516796 21049 log.cc:824] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:41.519341 21050 tablet_bootstrap.cc:654] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:41.520267 21049 tablet_bootstrap.cc:492] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: No bootstrap required, opened a new log
I20260620 02:22:41.520598 21049 ts_tablet_manager.cc:1397] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 02:22:41.520918 21050 log.cc:824] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:41.523265 21052 tablet_bootstrap.cc:492] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Bootstrap starting.
I20260620 02:22:41.524513 21050 tablet_bootstrap.cc:492] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: No bootstrap required, opened a new log
I20260620 02:22:41.524824 21050 ts_tablet_manager.cc:1397] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:22:41.527086 21052 tablet_bootstrap.cc:654] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Neither blocks nor log segments found. Creating new log.
I20260620 02:22:41.528383 21052 log.cc:824] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Log is configured to *not* fsync() on all Append() calls
I20260620 02:22:41.531253 21049 raft_consensus.cc:348] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.531716 21049 raft_consensus.cc:374] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:41.531754 21052 tablet_bootstrap.cc:492] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: No bootstrap required, opened a new log
I20260620 02:22:41.531839 21049 raft_consensus.cc:729] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c25b523e644312ae31d1e6ae8a097f, State: Initialized, Role: FOLLOWER
I20260620 02:22:41.532035 21052 ts_tablet_manager.cc:1397] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 02:22:41.532557 21050 raft_consensus.cc:348] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.532572 21049 consensus_queue.cc:260] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.532892 21050 raft_consensus.cc:374] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:41.532997 21050 raft_consensus.cc:729] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107df6b298774e178197728cadb9b515, State: Initialized, Role: FOLLOWER
I20260620 02:22:41.533787 21030 heartbeater.cc:503] Master 127.9.231.126:33713 was elected leader, sending a full tablet report...
I20260620 02:22:41.533715 21050 consensus_queue.cc:260] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.534399 21049 ts_tablet_manager.cc:1428] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260620 02:22:41.537113 21050 ts_tablet_manager.cc:1428] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 02:22:41.542030 21052 raft_consensus.cc:348] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.542399 21052 raft_consensus.cc:374] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:22:41.542470 21052 raft_consensus.cc:729] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3932ff7e9a84d9c88258b288b851a23, State: Initialized, Role: FOLLOWER
I20260620 02:22:41.543026 21052 consensus_queue.cc:260] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.544067 20896 heartbeater.cc:503] Master 127.9.231.126:33713 was elected leader, sending a full tablet report...
I20260620 02:22:41.544760 21052 ts_tablet_manager.cc:1428] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260620 02:22:41.569314 20763 tablet.cc:2369] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:22:41.600023 20897 tablet.cc:2369] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:22:41.602939 21055 raft_consensus.cc:482] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:22:41.603294 21055 raft_consensus.cc:504] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.605036 21055 leader_election.cc:288] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 107df6b298774e178197728cadb9b515 (127.9.231.65:40499), d3932ff7e9a84d9c88258b288b851a23 (127.9.231.66:34409)
I20260620 02:22:41.610805 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.605421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:40499 (local address 127.9.231.67:48545)
0620 02:22:41.605646 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:41.605669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:41.605806 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:41.606136 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:41.606141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:41.606168 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:41.606374 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.606383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.607534 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.607541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.608930 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.608942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.610063 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.610068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.610252 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.610275 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:22:41.610375 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 02:22:41.610508 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260620 02:22:41.610785 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.605403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:48545 (local address 127.9.231.65:40499)
0620 02:22:41.605789 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.605795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.605817 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:22:41.605936 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.605941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.605998 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.606107 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:41.606521 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.607415 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.609093 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.609881 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.610088 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.610167 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.610409 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.610534 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260620 02:22:41.611711 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.605632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:51229 (local address 127.9.231.66:34409)
0620 02:22:41.605882 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.605889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.606539 (+ 650us) server_negotiation.cc:407] Connection header received
0620 02:22:41.606726 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.606735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.606840 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.606935 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:22:41.607503 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.608293 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.609969 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.610735 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.610912 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.610979 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.611337 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.611464 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:22:41.611761 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.605533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:34409 (local address 127.9.231.67:51229)
0620 02:22:41.606431 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 02:22:41.606449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:22:41.606586 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:22:41.606959 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:22:41.606963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:22:41.606987 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:22:41.607317 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.607326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.608445 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.608452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.609810 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:22:41.609822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.610936 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.610942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:22:41.611134 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.611164 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:22:41.611294 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:22:41.611422 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":590,"threads_started":1}
I20260620 02:22:41.611922 20717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15cb0b64944a59aec608722db2b066" candidate_uuid: "c4c25b523e644312ae31d1e6ae8a097f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "107df6b298774e178197728cadb9b515" is_pre_election: true
I20260620 02:22:41.612515 20717 raft_consensus.cc:2454] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c25b523e644312ae31d1e6ae8a097f in term 0.
I20260620 02:22:41.612773 20851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15cb0b64944a59aec608722db2b066" candidate_uuid: "c4c25b523e644312ae31d1e6ae8a097f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3932ff7e9a84d9c88258b288b851a23" is_pre_election: true
I20260620 02:22:41.613211 20851 raft_consensus.cc:2454] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c25b523e644312ae31d1e6ae8a097f in term 0.
I20260620 02:22:41.613242 20919 leader_election.cc:302] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107df6b298774e178197728cadb9b515, c4c25b523e644312ae31d1e6ae8a097f; no voters:
I20260620 02:22:41.613606 21055 raft_consensus.cc:2781] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:22:41.613739 21055 raft_consensus.cc:482] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:22:41.613857 21055 raft_consensus.cc:3037] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:41.616220 21055 raft_consensus.cc:504] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.616938 21055 leader_election.cc:288] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [CANDIDATE]: Term 1 election: Requested vote from peers 107df6b298774e178197728cadb9b515 (127.9.231.65:40499), d3932ff7e9a84d9c88258b288b851a23 (127.9.231.66:34409)
I20260620 02:22:41.617380 20717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15cb0b64944a59aec608722db2b066" candidate_uuid: "c4c25b523e644312ae31d1e6ae8a097f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "107df6b298774e178197728cadb9b515"
I20260620 02:22:41.617497 20851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15cb0b64944a59aec608722db2b066" candidate_uuid: "c4c25b523e644312ae31d1e6ae8a097f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3932ff7e9a84d9c88258b288b851a23"
I20260620 02:22:41.617595 20717 raft_consensus.cc:3037] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:41.617743 20851 raft_consensus.cc:3037] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:22:41.619918 20717 raft_consensus.cc:2454] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c25b523e644312ae31d1e6ae8a097f in term 1.
I20260620 02:22:41.620376 20919 leader_election.cc:302] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107df6b298774e178197728cadb9b515, c4c25b523e644312ae31d1e6ae8a097f; no voters:
I20260620 02:22:41.620431 20851 raft_consensus.cc:2454] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c25b523e644312ae31d1e6ae8a097f in term 1.
I20260620 02:22:41.620687 21055 raft_consensus.cc:2781] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:22:41.621176 21055 raft_consensus.cc:686] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 1 LEADER]: Becoming Leader. State: Replica: c4c25b523e644312ae31d1e6ae8a097f, State: Running, Role: LEADER
I20260620 02:22:41.621729 21055 consensus_queue.cc:237] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } }
I20260620 02:22:41.626452 20572 catalog_manager.cc:5495] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f reported cstate change: term changed from 0 to 1, leader changed from <none> to c4c25b523e644312ae31d1e6ae8a097f (127.9.231.67). New cstate: current_term: 1 leader_uuid: "c4c25b523e644312ae31d1e6ae8a097f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c25b523e644312ae31d1e6ae8a097f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 } health_report { overall_health: UNKNOWN } } }
W20260620 02:22:41.632056 21031 tablet.cc:2369] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:22:41.677234 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.672171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.9.231.67:39907)
0620 02:22:41.672371 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.672375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.672392 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:22:41.672479 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.672482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.672562 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.672690 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:22:41.673045 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.673786 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.675163 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.675548 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.675640 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.676077 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:22:41.676614 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:22:41.676732 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.676841 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.676909 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:22:41.685210 20851 raft_consensus.cc:1261] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 1 FOLLOWER]: Refusing update from remote peer c4c25b523e644312ae31d1e6ae8a097f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:41.685200 20717 raft_consensus.cc:1261] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 1 FOLLOWER]: Refusing update from remote peer c4c25b523e644312ae31d1e6ae8a097f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:22:41.686199 21059 consensus_queue.cc:1035] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [LEADER]: Connected to new peer: Peer: permanent_uuid: "107df6b298774e178197728cadb9b515" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:41.686539 21055 consensus_queue.cc:1035] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3932ff7e9a84d9c88258b288b851a23" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:22:41.699175 21062 mvcc.cc:205] Tried to move back new op lower bound from 7298753174252744704 to 7298753174006915072. Current Snapshot: MvccSnapshot[applied={T|T < 7298753174252744704}]
I20260620 02:22:41.704438 21065 mvcc.cc:205] Tried to move back new op lower bound from 7298753174252744704 to 7298753174006915072. Current Snapshot: MvccSnapshot[applied={T|T < 7298753174252744704}]
I20260620 02:22:41.714041 20572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:41.714342 20572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:22:41.718216 20572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88cc650c3328436ab2542d38dd11c8f8: Sending DeleteTablet for 3 replicas of tablet da15cb0b64944a59aec608722db2b066
I20260620 02:22:41.719555 20965 tablet_service.cc:1514] Processing DeleteTablet for tablet da15cb0b64944a59aec608722db2b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:41 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:59564
I20260620 02:22:41.719642 20697 tablet_service.cc:1514] Processing DeleteTablet for tablet da15cb0b64944a59aec608722db2b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:41 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:38770
I20260620 02:22:41.719697 20831 tablet_service.cc:1514] Processing DeleteTablet for tablet da15cb0b64944a59aec608722db2b066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:22:41 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:46446
I20260620 02:22:41.721302 21073 tablet_replica.cc:330] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: stopping tablet replica
I20260620 02:22:41.721879 21073 raft_consensus.cc:2229] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:41.722503 21073 raft_consensus.cc:2258] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:41.724530 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20634
I20260620 02:22:41.724953 21073 ts_tablet_manager.cc:1905] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:41.732758 21073 ts_tablet_manager.cc:1918] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:41.733033 21073 log.cc:1196] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/da15cb0b64944a59aec608722db2b066
I20260620 02:22:41.733651 21073 ts_tablet_manager.cc:1939] T da15cb0b64944a59aec608722db2b066 P d3932ff7e9a84d9c88258b288b851a23: Deleting consensus metadata
I20260620 02:22:41.735826 20559 catalog_manager.cc:4841] TS d3932ff7e9a84d9c88258b288b851a23 (127.9.231.66:34409): tablet da15cb0b64944a59aec608722db2b066 (table test-table [id=3ecb6d104fda485ea597e6ec30604c11]) successfully deleted
I20260620 02:22:41.737485 21072 tablet_replica.cc:330] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: stopping tablet replica
I20260620 02:22:41.737744 21071 tablet_replica.cc:330] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: stopping tablet replica
I20260620 02:22:41.738071 21072 raft_consensus.cc:2229] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:22:41.738340 21071 raft_consensus.cc:2229] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:22:41.738663 21072 raft_consensus.cc:2258] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:41.739431 21071 raft_consensus.cc:2258] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:22:41.741434 21072 ts_tablet_manager.cc:1905] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:22:41.741501 21071 ts_tablet_manager.cc:1905] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:22:41 dist-test-slave-9vg1 krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:22:41.749100 21072 ts_tablet_manager.cc:1918] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:41.749251 21071 ts_tablet_manager.cc:1918] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:22:41.749470 21071 log.cc:1196] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/da15cb0b64944a59aec608722db2b066
I20260620 02:22:41.749313 21072 log.cc:1196] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/da15cb0b64944a59aec608722db2b066
I20260620 02:22:41.750056 21071 ts_tablet_manager.cc:1939] T da15cb0b64944a59aec608722db2b066 P c4c25b523e644312ae31d1e6ae8a097f: Deleting consensus metadata
I20260620 02:22:41.750252 21072 ts_tablet_manager.cc:1939] T da15cb0b64944a59aec608722db2b066 P 107df6b298774e178197728cadb9b515: Deleting consensus metadata
I20260620 02:22:41.752167 20560 catalog_manager.cc:4841] TS c4c25b523e644312ae31d1e6ae8a097f (127.9.231.67:39907): tablet da15cb0b64944a59aec608722db2b066 (table test-table [id=3ecb6d104fda485ea597e6ec30604c11]) successfully deleted
I20260620 02:22:41.752367 20558 catalog_manager.cc:4841] TS 107df6b298774e178197728cadb9b515 (127.9.231.65:40499): tablet da15cb0b64944a59aec608722db2b066 (table test-table [id=3ecb6d104fda485ea597e6ec30604c11]) successfully deleted
I20260620 02:22:41.755712 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.726096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.9.231.65:40499)
0620 02:22:41.726334 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.726343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.736766 (+ 10423us) server_negotiation.cc:407] Connection header received
0620 02:22:41.737026 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.737035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.737193 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.737317 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:41.739248 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.740420 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.742236 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.742909 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.743015 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.748518 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:41.748533 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:41.748552 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:41.748595 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:41.752823 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.753763 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.753771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.753773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.753897 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.754285 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.754288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.754290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.754633 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:41.754884 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.755255 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.755412 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:22:41.756431 20757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:41.849499 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20634
I20260620 02:22:41.874523 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20768
Jun 20 02:22:41 dist-test-slave-9vg1 krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:22:41.895116 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:41.875409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.9.231.66:34409)
0620 02:22:41.875598 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:41.875602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:41.875640 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:22:41.875905 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:41.875910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:41.875991 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:41.876083 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:41.877915 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.878669 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.880774 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:41.881162 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:41.881261 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:41.886757 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:41.886780 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:41.886803 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:41.886860 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:41.891292 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.892438 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.892475 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.892479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.892657 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:41.893110 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:41.893115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:41.893119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:41.893732 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:41.894061 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:41.894361 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 02:22:41.894620 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:22:41.895882 20891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:42.009070 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20768
I20260620 02:22:42.027212 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20902
Jun 20 02:22:42 dist-test-slave-9vg1 krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:22:42.045007 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:22:42.028103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.9.231.67:39907)
0620 02:22:42.028357 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:22:42.028364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:22:42.028389 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:22:42.028570 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:22:42.028576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:22:42.028667 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:22:42.028799 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:22:42.030165 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:42.031205 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:42.032521 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:22:42.033094 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:22:42.033259 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:22:42.038565 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:22:42.038585 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:22:42.038602 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:22:42.038654 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:22:42.041876 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:42.042826 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:42.042837 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:42.042839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:42.042998 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:22:42.043460 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:22:42.043464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:22:42.043466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:22:42.043812 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:22:42.044035 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:22:42.044472 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 02:22:42.044668 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 02:22:42.045840 21025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:42.170835 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20902
I20260620 02:22:42.191159 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 20541
I20260620 02:22:42.192471 20603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:22:42.306361 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 20541
2026-06-20T02:22:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4489 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:22:42 dist-test-slave-9vg1 krb5kdc[21081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:22:42 dist-test-slave-9vg1 krb5kdc[21081](info): set up 2 sockets
Jun 20 02:22:42 dist-test-slave-9vg1 krb5kdc[21081](info): commencing operation
krb5kdc: starting...
W20260620 02:22:43.442487 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:22:43 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:22:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/rangeradmin_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/rangeradmin_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/rangerlookup_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/rangerlookup_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
I20260620 02:22:43.802837 10141 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4D_H_y/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/postgres -l logfile start
2026-06-20 02:22:46.662 UTC [21111] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:22:46.662 UTC [21111] LOG: listening on IPv4 address "127.9.231.84", port 53785
2026-06-20 02:22:46.666 UTC [21111] LOG: listening on Unix socket "/tmp/.s.PGSQL.53785"
2026-06-20 02:22:46.691 UTC [21114] LOG: database system was shut down at 2026-06-20 02:22:45 UTC
2026-06-20 02:22:46.695 UTC [21111] LOG: database system is ready to accept connections
I20260620 02:22:47.680925 10141 mini_postgres.cc:96] Postgres bound to 53785
2026-06-20 02:22:47.700 UTC [21122] FATAL: database "slave" does not exist
127.9.231.84:53785 - accepting connections
I20260620 02:22:47.701548 10141 mini_ranger.cc:161] Starting Ranger...
I20260620 02:22:47.731334 10141 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:22:48.340816 10141 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:22:48.340987 10141 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-admin
2026-06-20 02:22:48,893 [I] DB FLAVOR :POSTGRES
2026-06-20 02:22:48,894 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:22:48,894 [I] Checking connection..
2026-06-20 02:22:48,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:22:49,171 [I] Checking connection passed.
2026-06-20 02:22:49,171 [I] --------- Verifying version history table ---------
2026-06-20 02:22:49,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:22:49,421 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:22:49,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:22:49,709 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:22:49,709 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:22:49,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:22:49.966 UTC [21186] WARNING: there is no transaction in progress
2026-06-20 02:22:49,973 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:22:49,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:22:50,252 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:22:50,252 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:22:50,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:22:50,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:22:50,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9vg1.c.gcp-upstream.internal','N') ;"
2026-06-20 02:22:51,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:22:51,297 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:22:51,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:22:51,575 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:22:51,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:22:51,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:22:52,089 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:22:52,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:22:53.132 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.149 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.168 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.186 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.203 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.223 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.286 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.295 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.312 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.324 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.335 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.350 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.361 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.369 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.377 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:53.385 UTC [21294] WARNING: there is no transaction in progress
2026-06-20 02:22:54,159 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:22:54,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:22:54,426 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:22:54,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:22:54,710 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:22:54,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:22:54,957 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:22:54,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:22:55,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:22:55,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9vg1.c.gcp-upstream.internal';"
2026-06-20 02:22:55,752 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:22:55,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:22:56,005 [I] DB_PATCHES have already been applied
I20260620 02:22:56.011382 10141 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-admin:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4D_H_y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:22:56.011554 10141 mini_ranger.cc:193] Using host: 127.9.231.84
I20260620 02:22:56.028636 10141 mini_ranger.cc:227] Ranger admin URL: http://127.9.231.84:58775
Jun 20, 2026 2:22:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:22:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:22:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:22:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:22:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:22:57.215903 10141 mini_ranger.cc:160] Time spent starting Ranger: real 9.514s user 0.000s sys 0.045s
Jun 20, 2026 2:23:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:23:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:23:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:23:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:23:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:23:20.708317 10141 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/rangerkms_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/rangerkms_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.231.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.231.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:23:20 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922200, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:23:20.901209 10141 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:23:20.923799 10141 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:23:21.859788 10141 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:23:21.859915 10141 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms
I20260620 02:23:21.860095 10141 mini_ranger_kms.cc:227] Using postgres at 127.9.231.84:53785
2026-06-20 02:23:22,734 [I] DB FLAVOR :POSTGRES
2026-06-20 02:23:22,734 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:23:22,734 [I] Checking connection
2026-06-20 02:23:22,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:23:23,016 [I] connection success
2026-06-20 02:23:23,016 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:23:23,016 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:23:23,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:23:23,262 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:23:23,263 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:23:23,263 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:23:23,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:53785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:23:23,531 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:23:24.078956 10141 mini_ranger_kms.cc:310] Created kms service
I20260620 02:23:24.281832 10141 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:23:24.330462 10141 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:23:24 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922200, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:23:24.714464 10141 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:23:24.714818 10141 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms:/tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:23:24.714875 10141 mini_ranger_kms.cc:240] Using host: 127.9.231.84
I20260620 02:23:24.733321 10141 mini_ranger_kms.cc:276] Ranger KMS PID: 21475
I20260620 02:23:24.733542 10141 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.231.84:47253
Jun 20, 2026 2:23:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:23:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:23:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:23:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:23:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:23:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 02:23:27 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.231.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:23:27.134100 10141 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.233s user 0.135s sys 0.409s
I20260620 02:23:27.134375 10141 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:23:27.134598 10141 mini_ranger_kms.cc:401] 127.9.231.84:47253/kms/v1/keys
Jun 20 02:23:27 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.231.84@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
Jun 20 02:23:27 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.231.84@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
Jun 20 02:23:30 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:23:30.300347 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:41785
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:41785
--ranger_config_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:47253/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:23:30.703156 21525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:23:30.703560 21525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:23:30.703725 21525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:23:30.711359 21525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:23:30.711525 21525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:23:30.711627 21525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:23:30.711728 21525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:23:30.711813 21525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:23:30.722308 21525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.231.84:47253/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:41785
--ranger_config_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:23:30.723775 21525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:23:30.725090 21525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:23:30.738265 21530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:23:30.738185 21531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:23:30.738679 21533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:23:30.740305 21525 server_base.cc:1034] running on GCE node
I20260620 02:23:30.741367 21525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:23:30.743175 21525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:23:30.744431 21525 hybrid_clock.cc:648] HybridClock initialized: now 1781922210744364 us; error 55 us; skew 500 ppm
I20260620 02:23:30.744920 21525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:23:30 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922210, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:23:30.754530 21525 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:23:30.756299 21525 webserver.cc:460] Webserver started at http://127.9.231.126:40259/ using document root <none> and password file <none>
I20260620 02:23:30.757018 21525 fs_manager.cc:359] Metadata directory not provided
I20260620 02:23:30.757162 21525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:23:30.757614 21525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:23:30 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922210, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:23:30.813777 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "62d18d07d36a484ca7109539ad74d5b3"
format_stamp: "Formatted at 2026-06-20 02:23:30 on dist-test-slave-9vg1"
server_key: "ebedb7fc0e65498293fd8ad382c3ac25"
server_key_iv: "bb4b3baba6b7bbc57d4e9e62c8a22668"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:30.814867 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "62d18d07d36a484ca7109539ad74d5b3"
format_stamp: "Formatted at 2026-06-20 02:23:30 on dist-test-slave-9vg1"
server_key: "ebedb7fc0e65498293fd8ad382c3ac25"
server_key_iv: "bb4b3baba6b7bbc57d4e9e62c8a22668"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:30.822304 21525 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260620 02:23:30.840354 21540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:30.841960 21525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:23:30.842176 21525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "62d18d07d36a484ca7109539ad74d5b3"
format_stamp: "Formatted at 2026-06-20 02:23:30 on dist-test-slave-9vg1"
server_key: "ebedb7fc0e65498293fd8ad382c3ac25"
server_key_iv: "bb4b3baba6b7bbc57d4e9e62c8a22668"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:30.842429 21525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:23:30.886703 21525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:23:30.891484 21525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:23:30.891837 21525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:23:30.916141 21525 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:41785
I20260620 02:23:30.916296 21591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:41785 every 8 connection(s)
I20260620 02:23:30.917778 21525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:23:30.923105 21592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:23:30.924777 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 21525
I20260620 02:23:30.925012 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:23:30.925652 10141 external_mini_cluster.cc:1428] Setting key c1c79dd6244f63a8b9d7a0f9a8e9860f
I20260620 02:23:30.938529 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3: Bootstrap starting.
Jun 20 02:23:30 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:23:30.942675 21592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3: Neither blocks nor log segments found. Creating new log.
I20260620 02:23:30.944463 21592 log.cc:824] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:23:30.947836 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3: No bootstrap required, opened a new log
I20260620 02:23:30.950582 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:30.927993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.9.231.126:41785)
0620 02:23:30.928972 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:30.928984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:30.929073 (+ 89us) server_negotiation.cc:407] Connection header received
0620 02:23:30.931037 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:30.931140 (+ 103us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:30.931621 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:30.932098 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:30.933498 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:30.934914 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:30.936321 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:30.937295 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:30.937531 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:30.941852 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:30.941862 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:30.941880 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:30.941926 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:30.946091 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:30.946925 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:30.946933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:30.946942 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:30.947174 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:30.947596 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:30.947602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:30.947605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:30.948265 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:30.948584 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:30.949067 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 02:23:30.949506 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":435,"threads_started":1}
I20260620 02:23:30.959120 21592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d18d07d36a484ca7109539ad74d5b3" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41785 } }
I20260620 02:23:30.959681 21592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:23:30.959805 21592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62d18d07d36a484ca7109539ad74d5b3, State: Initialized, Role: FOLLOWER
I20260620 02:23:30.960608 21592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d18d07d36a484ca7109539ad74d5b3" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41785 } }
I20260620 02:23:30.961086 21592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:23:30.961257 21592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:23:30.961442 21592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:23:30.965301 21592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d18d07d36a484ca7109539ad74d5b3" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41785 } }
I20260620 02:23:30.966064 21592 leader_election.cc:302] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62d18d07d36a484ca7109539ad74d5b3; no voters:
I20260620 02:23:30.966859 21592 leader_election.cc:288] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:23:30.966977 21597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:23:30.968683 21597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [term 1 LEADER]: Becoming Leader. State: Replica: 62d18d07d36a484ca7109539ad74d5b3, State: Running, Role: LEADER
I20260620 02:23:30.969372 21597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d18d07d36a484ca7109539ad74d5b3" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41785 } }
I20260620 02:23:30.969609 21592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:23:30.972152 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62d18d07d36a484ca7109539ad74d5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d18d07d36a484ca7109539ad74d5b3" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41785 } } }
I20260620 02:23:30.972584 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:23:30.973269 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62d18d07d36a484ca7109539ad74d5b3. Latest consensus state: current_term: 1 leader_uuid: "62d18d07d36a484ca7109539ad74d5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d18d07d36a484ca7109539ad74d5b3" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41785 } } }
I20260620 02:23:30.973850 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:23:30.974370 21603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:23:30.986529 21603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:23:30.997551 21592 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:23:30.998605 21603 catalog_manager.cc:1347] Generated new cluster ID: dd6a01be2aa94a2da690c7fad2b1a9b4
I20260620 02:23:30.998723 21603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:23:31.025578 21603 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:23:31.027380 21603 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:23:31.046653 21603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3: Generated new TSK 0
I20260620 02:23:31.048074 21603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:23:31.876396 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:47253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:41785
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:23:32 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:23:32.357630 21637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:23:32.358156 21637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:23:32.358368 21637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:23:32.370615 21637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:23:32.370839 21637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:23:32.371111 21637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:23:32.387732 21637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.231.84:47253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:23:32.389748 21637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:23:32.391670 21637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:23:32.415169 21648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:23:32.415462 21647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:23:32.421080 21637 server_base.cc:1034] running on GCE node
W20260620 02:23:32.421941 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:23:32.422966 21637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:23:32.426736 21637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:23:32.428224 21637 hybrid_clock.cc:648] HybridClock initialized: now 1781922212428076 us; error 132 us; skew 500 ppm
I20260620 02:23:32.428679 21637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:23:32 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:23:32.435427 21637 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:23:32.438023 21637 webserver.cc:460] Webserver started at http://127.9.231.65:42639/ using document root <none> and password file <none>
I20260620 02:23:32.439013 21637 fs_manager.cc:359] Metadata directory not provided
I20260620 02:23:32.439272 21637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:23:32.439767 21637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:23:32 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:23:32.459854 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "0439aaa6d0e14805ad72496e191192a2"
format_stamp: "Formatted at 2026-06-20 02:23:32 on dist-test-slave-9vg1"
server_key: "bfc10f96491727ca3b33d8c719b6bb35"
server_key_iv: "43d8c3136185b0b9cdd9f0308c2fbd94"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:32.460764 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "0439aaa6d0e14805ad72496e191192a2"
format_stamp: "Formatted at 2026-06-20 02:23:32 on dist-test-slave-9vg1"
server_key: "bfc10f96491727ca3b33d8c719b6bb35"
server_key_iv: "43d8c3136185b0b9cdd9f0308c2fbd94"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:32.466622 21637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:23:32.499819 21657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:32.512812 21637 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.005s sys 0.009s
I20260620 02:23:32.513182 21637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "0439aaa6d0e14805ad72496e191192a2"
format_stamp: "Formatted at 2026-06-20 02:23:32 on dist-test-slave-9vg1"
server_key: "bfc10f96491727ca3b33d8c719b6bb35"
server_key_iv: "43d8c3136185b0b9cdd9f0308c2fbd94"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:32.513630 21637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:23:32.584229 21637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:23:32.590864 21637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:23:32.591380 21637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:23:32.593053 21637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:23:32.595623 21637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:23:32.595808 21637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 02:23:32.596026 21637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:23:32.596149 21637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:23:32 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:23:32.738101 21637 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:36879
I20260620 02:23:32.738186 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:36879 every 8 connection(s)
I20260620 02:23:32.739933 21637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:23:32.744676 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 21637
I20260620 02:23:32.744897 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:23:32.745766 10141 external_mini_cluster.cc:1428] Setting key 95eb25bc633d0de01119f2ed339c911f
Jun 20 02:23:32 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
I20260620 02:23:32.794543 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:32.756319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:57975 (local address 127.9.231.126:41785)
0620 02:23:32.768981 (+ 12662us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:32.768990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:32.769026 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:23:32.770247 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:32.770257 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:32.770370 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:32.770492 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:32.773082 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:32.774412 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:32.777725 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:32.778533 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:32.778665 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:32.784181 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:32.784189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:32.784194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:32.784241 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:32.788283 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:32.789299 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:32.789307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:32.789309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:32.789458 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:32.789982 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:32.789993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:32.789996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:32.790417 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:32.790596 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:32.793664 (+ 3068us) server_negotiation.cc:299] Negotiation successful
0620 02:23:32.793975 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12486,"thread_start_us":14755,"threads_started":1}
I20260620 02:23:32.796047 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:32.757062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:41785 (local address 127.9.231.65:57975)
0620 02:23:32.765523 (+ 8461us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:32.765593 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:32.769694 (+ 4101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:32.771046 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:32.771106 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:32.771839 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:23:32.772795 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:32.772829 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:32.774650 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:32.774660 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:32.777492 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:32.777514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:32.779129 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:32.779142 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:32.779505 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:32.780425 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:23:32.780473 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:23:32.783833 (+ 3360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:23:32.788515 (+ 4682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:23:32.788524 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:23:32.788539 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:23:32.789030 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:23:32.789609 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:23:32.789614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:23:32.789618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:23:32.789839 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:23:32.790570 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:23:32.790579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:23:32.791176 (+ 597us) client_negotiation.cc:769] Sending connection context
0620 02:23:32.791534 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 02:23:32.792181 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8090,"thread_start_us":7737,"threads_started":1}
I20260620 02:23:32.798597 21770 heartbeater.cc:347] Connected to a master server at 127.9.231.126:41785
I20260620 02:23:32.798951 21770 heartbeater.cc:464] Registering TS with master...
I20260620 02:23:32.799932 21770 heartbeater.cc:511] Master 127.9.231.126:41785 requested a full tablet report, sending...
I20260620 02:23:32.802660 21557 ts_manager.cc:194] Registered new tserver with Master: 0439aaa6d0e14805ad72496e191192a2 (127.9.231.65:36879)
I20260620 02:23:32.804785 21557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:57975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:23:32.831599 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:47253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:41785
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:23:33 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
W20260620 02:23:33.367731 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:23:33.368103 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:23:33.368275 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:23:33.376981 21776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:23:33.377279 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:23:33.377517 21776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:23:33.391795 21776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.231.84:47253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:23:33.393486 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:23:33.395022 21776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:23:33.412681 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:23:33.413039 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:23:33.414135 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:23:33.416052 21776 server_base.cc:1034] running on GCE node
I20260620 02:23:33.417088 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:23:33.418655 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:23:33.419888 21776 hybrid_clock.cc:648] HybridClock initialized: now 1781922213419804 us; error 88 us; skew 500 ppm
I20260620 02:23:33.420254 21776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:23:33 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922213, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:23:33.424443 21776 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:23:33.426556 21776 webserver.cc:460] Webserver started at http://127.9.231.66:36451/ using document root <none> and password file <none>
I20260620 02:23:33.427640 21776 fs_manager.cc:359] Metadata directory not provided
I20260620 02:23:33.427886 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:23:33.428347 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:23:33 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922213, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:23:33.447860 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "c4ba07cceee543f1aabb5edef5abb6d5"
format_stamp: "Formatted at 2026-06-20 02:23:33 on dist-test-slave-9vg1"
server_key: "7c4839316051a7881e57ebff0f6e668a"
server_key_iv: "9104e793573a52bd5cd70ed853065e52"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:33.448916 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "c4ba07cceee543f1aabb5edef5abb6d5"
format_stamp: "Formatted at 2026-06-20 02:23:33 on dist-test-slave-9vg1"
server_key: "7c4839316051a7881e57ebff0f6e668a"
server_key_iv: "9104e793573a52bd5cd70ed853065e52"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:33.456442 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 02:23:33.487224 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:33.489588 21776 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.008s sys 0.006s
I20260620 02:23:33.489877 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "c4ba07cceee543f1aabb5edef5abb6d5"
format_stamp: "Formatted at 2026-06-20 02:23:33 on dist-test-slave-9vg1"
server_key: "7c4839316051a7881e57ebff0f6e668a"
server_key_iv: "9104e793573a52bd5cd70ed853065e52"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:33.490242 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:23:33.524180 21776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:23:33.529819 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:23:33.530201 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:23:33.531467 21776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:23:33.533314 21776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:23:33.533423 21776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:33.533607 21776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:23:33.533694 21776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:33.600065 21776 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:44045
I20260620 02:23:33.600107 21912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:44045 every 8 connection(s)
I20260620 02:23:33.601876 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:23:33.604595 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 21776
I20260620 02:23:33.604763 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:23:33.605281 10141 external_mini_cluster.cc:1428] Setting key 5662131b4a7b8da2347dc1d525444ca0
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Jun 20 02:23:33 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922213, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:23:33.664541 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:33.615904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:41785 (local address 127.9.231.66:40975)
0620 02:23:33.623836 (+ 7932us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:33.623895 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:33.626697 (+ 2802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:33.627976 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:33.627990 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:33.628621 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:23:33.629448 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:33.629473 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:33.631026 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:33.631033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:33.633198 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:33.633224 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:33.634513 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:33.634656 (+ 143us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:33.635007 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:33.646969 (+ 11962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:23:33.647027 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:23:33.651225 (+ 4198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:23:33.659255 (+ 8030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:23:33.659267 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:23:33.659299 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:23:33.659976 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:23:33.660485 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:23:33.660490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:23:33.660493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:23:33.660768 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:23:33.661505 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:23:33.661515 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:23:33.662151 (+ 636us) client_negotiation.cc:769] Sending connection context
0620 02:23:33.662470 (+ 319us) client_negotiation.cc:240] Negotiation successful
0620 02:23:33.663095 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7537,"thread_start_us":7398,"threads_started":1}
I20260620 02:23:33.666615 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:33.615385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:40975 (local address 127.9.231.126:41785)
0620 02:23:33.623592 (+ 8207us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:33.623599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:33.627189 (+ 3590us) server_negotiation.cc:407] Connection header received
0620 02:23:33.627283 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:33.627289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:33.627386 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:33.627490 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:33.629689 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:33.630810 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:33.633398 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:33.634012 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:33.634126 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:33.654983 (+ 20857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:33.654993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:33.654999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:33.655103 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:33.658958 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:33.660179 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:33.660185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:33.660189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:33.660348 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:33.660924 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:33.660929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:33.660933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:33.661349 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:33.665748 (+ 4399us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:33.665951 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 02:23:33.666235 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8047,"thread_start_us":7891,"threads_started":1}
I20260620 02:23:33.668352 21913 heartbeater.cc:347] Connected to a master server at 127.9.231.126:41785
I20260620 02:23:33.668696 21913 heartbeater.cc:464] Registering TS with master...
I20260620 02:23:33.669559 21913 heartbeater.cc:511] Master 127.9.231.126:41785 requested a full tablet report, sending...
I20260620 02:23:33.671509 21557 ts_manager.cc:194] Registered new tserver with Master: c4ba07cceee543f1aabb5edef5abb6d5 (127.9.231.66:44045)
I20260620 02:23:33.672601 21557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:40975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:23:33.685981 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:47253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:41785
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:23:33.809007 21770 heartbeater.cc:503] Master 127.9.231.126:41785 was elected leader, sending a full tablet report...
W20260620 02:23:34.131789 21919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:23:34.132230 21919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:23:34.132452 21919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:23:34.141045 21919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:23:34.141278 21919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:23:34.141485 21919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:23:34.152798 21919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.231.84:47253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:23:34.154156 21919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:23:34.155589 21919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:23:34.171034 21924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:23:34.171154 21925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:23:34.171314 21927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:23:34.172813 21919 server_base.cc:1034] running on GCE node
I20260620 02:23:34.173537 21919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:23:34.174635 21919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:23:34.175895 21919 hybrid_clock.cc:648] HybridClock initialized: now 1781922214175814 us; error 55 us; skew 500 ppm
I20260620 02:23:34.176327 21919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:23:34 dist-test-slave-9vg1 krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:23:34.179855 21919 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:23:34.181402 21919 webserver.cc:460] Webserver started at http://127.9.231.67:39751/ using document root <none> and password file <none>
I20260620 02:23:34.182121 21919 fs_manager.cc:359] Metadata directory not provided
I20260620 02:23:34.182269 21919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:23:34.182590 21919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:23:34 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:23:34.198177 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "c688cb118ccf409bb7faaadfd8207729"
format_stamp: "Formatted at 2026-06-20 02:23:34 on dist-test-slave-9vg1"
server_key: "df2cb48135a4585a4e356b63d4c47ac4"
server_key_iv: "6272fe95d26d20c3c993852b38950d3f"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:34.199049 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "c688cb118ccf409bb7faaadfd8207729"
format_stamp: "Formatted at 2026-06-20 02:23:34 on dist-test-slave-9vg1"
server_key: "df2cb48135a4585a4e356b63d4c47ac4"
server_key_iv: "6272fe95d26d20c3c993852b38950d3f"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:34.205284 21919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 02:23:34.221539 21934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:34.227334 21919 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260620 02:23:34.227715 21919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "c688cb118ccf409bb7faaadfd8207729"
format_stamp: "Formatted at 2026-06-20 02:23:34 on dist-test-slave-9vg1"
server_key: "df2cb48135a4585a4e356b63d4c47ac4"
server_key_iv: "6272fe95d26d20c3c993852b38950d3f"
server_key_version: "kuduclusterkey@0"
I20260620 02:23:34.228049 21919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:23:34.251257 21919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:23:34.255838 21919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:23:34.256178 21919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:23:34.257318 21919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:23:34.258873 21919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:23:34.258978 21919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:34.259166 21919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:23:34.259241 21919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:23:34.303398 21919 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:43497
I20260620 02:23:34.303526 22046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:43497 every 8 connection(s)
I20260620 02:23:34.304973 21919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:23:34.306128 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 21919
I20260620 02:23:34.306320 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:23:34.307013 10141 external_mini_cluster.cc:1428] Setting key f5069eab1f8e7270641f4149feee50ee
Jun 20 02:23:34 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:23:34.344724 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.308763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:45765 (local address 127.9.231.126:41785)
0620 02:23:34.321080 (+ 12317us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.321089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.321116 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:23:34.322480 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.322486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.322583 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.322725 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:34.324634 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.326088 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.328348 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.329471 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.329626 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.334642 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:34.334650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:34.334657 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:34.334710 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:34.340021 (+ 5311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:34.341422 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:34.341426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:34.341429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:34.341575 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:34.342358 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:34.342362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:34.342364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:34.342701 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:34.342861 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.344097 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.344309 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12157,"thread_start_us":11957,"threads_started":1}
I20260620 02:23:34.347018 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.309570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:41785 (local address 127.9.231.67:45765)
0620 02:23:34.319979 (+ 10409us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:34.320046 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:34.322155 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:34.322998 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:34.323008 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:34.323550 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:23:34.324407 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.324433 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.326579 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.326584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.328152 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.328179 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.329751 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.329757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.329986 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.330793 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:23:34.330820 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:23:34.334161 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:23:34.340202 (+ 6041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:23:34.340216 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:23:34.340243 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:23:34.341267 (+ 1024us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:23:34.341691 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:23:34.341698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:23:34.341705 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:23:34.342225 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:23:34.342841 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:23:34.342851 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:23:34.343497 (+ 646us) client_negotiation.cc:769] Sending connection context
0620 02:23:34.343924 (+ 427us) client_negotiation.cc:240] Negotiation successful
0620 02:23:34.344759 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10008,"thread_start_us":9715,"threads_started":1}
I20260620 02:23:34.349071 22047 heartbeater.cc:347] Connected to a master server at 127.9.231.126:41785
I20260620 02:23:34.349463 22047 heartbeater.cc:464] Registering TS with master...
I20260620 02:23:34.350354 22047 heartbeater.cc:511] Master 127.9.231.126:41785 requested a full tablet report, sending...
I20260620 02:23:34.352084 21557 ts_manager.cc:194] Registered new tserver with Master: c688cb118ccf409bb7faaadfd8207729 (127.9.231.67:43497)
I20260620 02:23:34.353024 21557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:45765
I20260620 02:23:34.366044 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:23:34.395130 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.374177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.9.231.126:41785)
0620 02:23:34.374415 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.374424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.375137 (+ 713us) server_negotiation.cc:407] Connection header received
0620 02:23:34.375406 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.375415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.375598 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.375751 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:34.377434 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.379435 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.380975 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.382230 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.382402 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.385254 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:34.385264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:34.385272 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:34.385329 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:34.391371 (+ 6042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:34.392245 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:34.392254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:34.392260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:34.392489 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:34.392899 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:34.392906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:34.392911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:34.393474 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:34.393875 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.394105 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.394444 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:23:34.401178 21557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:23:34.416821 21557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:23:34.451486 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.432361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:44045 (local address 127.0.0.1:35012)
0620 02:23:34.436228 (+ 3867us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:34.436275 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:34.436465 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:34.438343 (+ 1878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:34.438351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:34.438408 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:23:34.439264 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.439287 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.441794 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.441803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.445866 (+ 4063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.445890 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.450080 (+ 4190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.450088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.450482 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.450550 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 02:23:34.450658 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:23:34.450847 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3596,"thread_start_us":3450,"threads_started":1}
I20260620 02:23:34.452647 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.432636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.9.231.66:44045)
0620 02:23:34.434131 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.434138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.437184 (+ 3046us) server_negotiation.cc:407] Connection header received
0620 02:23:34.437314 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.437321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.437675 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.437967 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:23:34.439527 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.441605 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.446106 (+ 4501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.449445 (+ 3339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.449586 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.449814 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.451894 (+ 2080us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.452150 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":1175,"threads_started":1}
I20260620 02:23:34.457103 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.432825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:43497 (local address 127.0.0.1:42526)
0620 02:23:34.447140 (+ 14315us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:34.447214 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:34.447377 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:34.448219 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:34.448224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:34.448251 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:23:34.449133 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.449147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.451191 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.451197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.453854 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.453869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.456155 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.456161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.456343 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.456381 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:23:34.456471 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:23:34.456607 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14103,"spinlock_wait_cycles":1280,"thread_start_us":13874,"threads_started":1}
I20260620 02:23:34.458004 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.433590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.9.231.67:43497)
0620 02:23:34.435662 (+ 2072us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.435670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.447347 (+ 11677us) server_negotiation.cc:407] Connection header received
0620 02:23:34.447567 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.447581 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.447988 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.448229 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:23:34.449389 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.450994 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.454045 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.455593 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.455716 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.455957 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.457420 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.457606 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1889,"thread_start_us":1690,"threads_started":1}
I20260620 02:23:34.457638 21848 tablet_service.cc:1467] Processing CreateTablet for tablet 43762df6c897405281bc10c8be6e1076 (DEFAULT_TABLE table=test-table [id=80b47548f07f4cd080f02aebbc373a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:23:34.459493 21848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43762df6c897405281bc10c8be6e1076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:23:34.461836 21982 tablet_service.cc:1467] Processing CreateTablet for tablet 43762df6c897405281bc10c8be6e1076 (DEFAULT_TABLE table=test-table [id=80b47548f07f4cd080f02aebbc373a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:23:34.462852 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.438078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.9.231.65:36879)
0620 02:23:34.439112 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.439124 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.448305 (+ 9181us) server_negotiation.cc:407] Connection header received
0620 02:23:34.448674 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.448686 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.448950 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.449125 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:23:34.451897 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.454619 (+ 2722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.457110 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.458645 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.458764 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.458995 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.462184 (+ 3189us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.462374 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":545,"threads_started":1}
I20260620 02:23:34.463595 21982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43762df6c897405281bc10c8be6e1076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:23:34.464367 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.436855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:36879 (local address 127.0.0.1:60022)
0620 02:23:34.448153 (+ 11298us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:34.448181 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:34.448444 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:34.449464 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:34.449472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:34.449519 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:23:34.450175 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.450190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.454789 (+ 4599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.454797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.456906 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.456920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.459249 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.459255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.459433 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.459467 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:23:34.459562 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:23:34.463965 (+ 4403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11143,"mutex_wait_us":1,"thread_start_us":10808,"threads_started":1}
I20260620 02:23:34.468585 21705 tablet_service.cc:1467] Processing CreateTablet for tablet 43762df6c897405281bc10c8be6e1076 (DEFAULT_TABLE table=test-table [id=80b47548f07f4cd080f02aebbc373a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:23:34.470939 21705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43762df6c897405281bc10c8be6e1076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:23:34.488905 22066 tablet_bootstrap.cc:492] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Bootstrap starting.
I20260620 02:23:34.496232 22067 tablet_bootstrap.cc:492] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Bootstrap starting.
I20260620 02:23:34.497641 22066 tablet_bootstrap.cc:654] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Neither blocks nor log segments found. Creating new log.
I20260620 02:23:34.499847 22068 tablet_bootstrap.cc:492] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Bootstrap starting.
I20260620 02:23:34.500790 22067 tablet_bootstrap.cc:654] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Neither blocks nor log segments found. Creating new log.
I20260620 02:23:34.501289 22066 log.cc:824] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Log is configured to *not* fsync() on all Append() calls
I20260620 02:23:34.502316 22067 log.cc:824] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:23:34.504204 22068 tablet_bootstrap.cc:654] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Neither blocks nor log segments found. Creating new log.
I20260620 02:23:34.506059 22068 log.cc:824] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:23:34.511500 22068 tablet_bootstrap.cc:492] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: No bootstrap required, opened a new log
I20260620 02:23:34.511500 22067 tablet_bootstrap.cc:492] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: No bootstrap required, opened a new log
I20260620 02:23:34.511545 22066 tablet_bootstrap.cc:492] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: No bootstrap required, opened a new log
I20260620 02:23:34.511828 22068 ts_tablet_manager.cc:1397] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260620 02:23:34.511875 22066 ts_tablet_manager.cc:1397] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20260620 02:23:34.511873 22067 ts_tablet_manager.cc:1397] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260620 02:23:34.523792 22067 raft_consensus.cc:348] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.524510 22067 raft_consensus.cc:374] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:23:34.524240 22068 raft_consensus.cc:348] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.524686 22068 raft_consensus.cc:374] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:23:34.524823 22068 raft_consensus.cc:729] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0439aaa6d0e14805ad72496e191192a2, State: Initialized, Role: FOLLOWER
I20260620 02:23:34.524866 22067 raft_consensus.cc:729] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ba07cceee543f1aabb5edef5abb6d5, State: Initialized, Role: FOLLOWER
I20260620 02:23:34.525096 22066 raft_consensus.cc:348] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.525589 22066 raft_consensus.cc:374] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:23:34.525700 22066 raft_consensus.cc:729] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688cb118ccf409bb7faaadfd8207729, State: Initialized, Role: FOLLOWER
I20260620 02:23:34.525882 22067 consensus_queue.cc:260] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.526460 22066 consensus_queue.cc:260] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.527541 22068 consensus_queue.cc:260] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.531805 22068 ts_tablet_manager.cc:1428] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Time spent starting tablet: real 0.020s user 0.009s sys 0.007s
I20260620 02:23:34.543679 21913 heartbeater.cc:503] Master 127.9.231.126:41785 was elected leader, sending a full tablet report...
I20260620 02:23:34.544190 22067 ts_tablet_manager.cc:1428] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Time spent starting tablet: real 0.032s user 0.013s sys 0.017s
I20260620 02:23:34.544373 22066 ts_tablet_manager.cc:1428] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Time spent starting tablet: real 0.032s user 0.017s sys 0.015s
I20260620 02:23:34.544924 22047 heartbeater.cc:503] Master 127.9.231.126:41785 was elected leader, sending a full tablet report...
W20260620 02:23:34.563001 22048 tablet.cc:2369] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:23:34.610526 21914 tablet.cc:2369] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:23:34.755995 21771 tablet.cc:2369] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:23:34.779047 22072 raft_consensus.cc:482] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:23:34.779469 22072 raft_consensus.cc:504] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.781260 22072 leader_election.cc:288] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688cb118ccf409bb7faaadfd8207729 (127.9.231.67:43497), 0439aaa6d0e14805ad72496e191192a2 (127.9.231.65:36879)
I20260620 02:23:34.787981 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.781737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:54107 (local address 127.9.231.67:43497)
0620 02:23:34.781971 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.781980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.782715 (+ 735us) server_negotiation.cc:407] Connection header received
0620 02:23:34.782918 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.782922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.782981 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.783124 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:23:34.783698 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.784453 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.786218 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.786975 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.787212 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.787291 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.787588 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.787713 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:23:34.788003 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.781769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:57563 (local address 127.9.231.65:36879)
0620 02:23:34.782005 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.782013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.782814 (+ 801us) server_negotiation.cc:407] Connection header received
0620 02:23:34.783005 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.783009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.783122 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.783225 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:23:34.783836 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.784565 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.786239 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.786980 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.787221 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.787299 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.787605 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.787734 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:23:34.788128 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.781647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:43497 (local address 127.9.231.66:54107)
0620 02:23:34.782601 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:34.782616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:34.782754 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:34.783144 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:34.783148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:34.783172 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:23:34.783542 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.783551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.784594 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.784601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.786059 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.786070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.787215 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.787222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.787390 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.787454 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 02:23:34.787563 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 02:23:34.787711 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":19,"thread_start_us":743,"threads_started":1}
I20260620 02:23:34.788408 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.781657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:36879 (local address 127.9.231.66:57563)
0620 02:23:34.782702 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 02:23:34.782721 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:23:34.782860 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:23:34.783242 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:23:34.783246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:23:34.783284 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:23:34.783677 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.783685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.784697 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.784702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.786091 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:23:34.786102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.787256 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.787263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:23:34.787456 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.787479 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:23:34.787589 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:23:34.787713 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":25,"thread_start_us":582,"threads_started":1}
I20260620 02:23:34.789309 22002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43762df6c897405281bc10c8be6e1076" candidate_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688cb118ccf409bb7faaadfd8207729" is_pre_election: true
I20260620 02:23:34.789464 21725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43762df6c897405281bc10c8be6e1076" candidate_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0439aaa6d0e14805ad72496e191192a2" is_pre_election: true
I20260620 02:23:34.789925 22002 raft_consensus.cc:2454] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4ba07cceee543f1aabb5edef5abb6d5 in term 0.
I20260620 02:23:34.790051 21725 raft_consensus.cc:2454] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4ba07cceee543f1aabb5edef5abb6d5 in term 0.
I20260620 02:23:34.790604 21803 leader_election.cc:302] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4ba07cceee543f1aabb5edef5abb6d5, c688cb118ccf409bb7faaadfd8207729; no voters:
I20260620 02:23:34.791021 22072 raft_consensus.cc:2781] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:23:34.791244 22072 raft_consensus.cc:482] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:23:34.791373 22072 raft_consensus.cc:3037] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:23:34.793543 22072 raft_consensus.cc:504] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.794138 22072 leader_election.cc:288] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [CANDIDATE]: Term 1 election: Requested vote from peers c688cb118ccf409bb7faaadfd8207729 (127.9.231.67:43497), 0439aaa6d0e14805ad72496e191192a2 (127.9.231.65:36879)
I20260620 02:23:34.794709 22002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43762df6c897405281bc10c8be6e1076" candidate_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688cb118ccf409bb7faaadfd8207729"
I20260620 02:23:34.794709 21725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43762df6c897405281bc10c8be6e1076" candidate_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0439aaa6d0e14805ad72496e191192a2"
I20260620 02:23:34.794934 22002 raft_consensus.cc:3037] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:23:34.794934 21725 raft_consensus.cc:3037] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:23:34.798193 21725 raft_consensus.cc:2454] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ba07cceee543f1aabb5edef5abb6d5 in term 1.
I20260620 02:23:34.798193 22002 raft_consensus.cc:2454] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ba07cceee543f1aabb5edef5abb6d5 in term 1.
I20260620 02:23:34.798658 21804 leader_election.cc:302] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0439aaa6d0e14805ad72496e191192a2, c4ba07cceee543f1aabb5edef5abb6d5; no voters:
I20260620 02:23:34.799095 22072 raft_consensus.cc:2781] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:23:34.799503 22072 raft_consensus.cc:686] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 1 LEADER]: Becoming Leader. State: Replica: c4ba07cceee543f1aabb5edef5abb6d5, State: Running, Role: LEADER
I20260620 02:23:34.800046 22072 consensus_queue.cc:237] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } }
I20260620 02:23:34.804659 21556 catalog_manager.cc:5495] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4ba07cceee543f1aabb5edef5abb6d5 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ba07cceee543f1aabb5edef5abb6d5" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 44045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 } health_report { overall_health: UNKNOWN } } }
I20260620 02:23:34.821131 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.815516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.9.231.66:44045)
0620 02:23:34.815764 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.815770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.815790 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:23:34.815975 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.815980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.816072 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.816225 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:23:34.816625 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.817357 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.818857 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.819328 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.819465 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.819976 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:23:34.820504 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:23:34.820661 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.820774 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.820857 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:23:34.829595 22002 raft_consensus.cc:1261] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 1 FOLLOWER]: Refusing update from remote peer c4ba07cceee543f1aabb5edef5abb6d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:23:34.829596 21725 raft_consensus.cc:1261] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 1 FOLLOWER]: Refusing update from remote peer c4ba07cceee543f1aabb5edef5abb6d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:23:34.830627 22077 consensus_queue.cc:1035] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0439aaa6d0e14805ad72496e191192a2" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:23:34.830994 22072 consensus_queue.cc:1035] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688cb118ccf409bb7faaadfd8207729" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 43497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:23:34.857692 21557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:23:34.858023 21557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:23:34.862288 21557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62d18d07d36a484ca7109539ad74d5b3: Sending DeleteTablet for 3 replicas of tablet 43762df6c897405281bc10c8be6e1076
I20260620 02:23:34.864841 21848 tablet_service.cc:1514] Processing DeleteTablet for tablet 43762df6c897405281bc10c8be6e1076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:23:34 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:35012
I20260620 02:23:34.865082 21705 tablet_service.cc:1514] Processing DeleteTablet for tablet 43762df6c897405281bc10c8be6e1076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:23:34 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:60022
I20260620 02:23:34.867655 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 21637
I20260620 02:23:34.872255 21982 tablet_service.cc:1514] Processing DeleteTablet for tablet 43762df6c897405281bc10c8be6e1076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:23:34 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:42526
I20260620 02:23:34.881831 22093 tablet_replica.cc:330] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: stopping tablet replica
I20260620 02:23:34.882506 22093 raft_consensus.cc:2229] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:23:34.883214 22093 raft_consensus.cc:2258] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:23:34.885483 22093 ts_tablet_manager.cc:1905] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:23:34.887518 22091 tablet_replica.cc:330] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: stopping tablet replica
I20260620 02:23:34.887995 22091 raft_consensus.cc:2229] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:23:34.888515 22091 raft_consensus.cc:2258] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:23:34.890698 22091 ts_tablet_manager.cc:1905] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:23:34.892608 22093 ts_tablet_manager.cc:1918] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:23:34.892792 22093 log.cc:1196] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/43762df6c897405281bc10c8be6e1076
I20260620 02:23:34.893318 22093 ts_tablet_manager.cc:1939] T 43762df6c897405281bc10c8be6e1076 P c688cb118ccf409bb7faaadfd8207729: Deleting consensus metadata
I20260620 02:23:34.895358 21543 catalog_manager.cc:4841] TS c688cb118ccf409bb7faaadfd8207729 (127.9.231.67:43497): tablet 43762df6c897405281bc10c8be6e1076 (table test-table [id=80b47548f07f4cd080f02aebbc373a90]) successfully deleted
I20260620 02:23:34.897297 22090 tablet_replica.cc:330] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: stopping tablet replica
I20260620 02:23:34.897420 22091 ts_tablet_manager.cc:1918] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:23:34.897598 22091 log.cc:1196] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/43762df6c897405281bc10c8be6e1076
I20260620 02:23:34.898008 22090 raft_consensus.cc:2229] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:23:34.898162 22091 ts_tablet_manager.cc:1939] T 43762df6c897405281bc10c8be6e1076 P 0439aaa6d0e14805ad72496e191192a2: Deleting consensus metadata
I20260620 02:23:34.899845 22090 raft_consensus.cc:2258] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:23:34.899896 21544 catalog_manager.cc:4841] TS 0439aaa6d0e14805ad72496e191192a2 (127.9.231.65:36879): tablet 43762df6c897405281bc10c8be6e1076 (table test-table [id=80b47548f07f4cd080f02aebbc373a90]) successfully deleted
I20260620 02:23:34.901985 22090 ts_tablet_manager.cc:1905] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:23:34 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:23:34.908548 22090 ts_tablet_manager.cc:1918] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:23:34.908757 22090 log.cc:1196] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/43762df6c897405281bc10c8be6e1076
I20260620 02:23:34.909390 22090 ts_tablet_manager.cc:1939] T 43762df6c897405281bc10c8be6e1076 P c4ba07cceee543f1aabb5edef5abb6d5: Deleting consensus metadata
I20260620 02:23:34.911247 21541 catalog_manager.cc:4841] TS c4ba07cceee543f1aabb5edef5abb6d5 (127.9.231.66:44045): tablet 43762df6c897405281bc10c8be6e1076 (table test-table [id=80b47548f07f4cd080f02aebbc373a90]) successfully deleted
I20260620 02:23:34.915910 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:34.869266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.9.231.65:36879)
0620 02:23:34.869458 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:34.869464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:34.893351 (+ 23887us) server_negotiation.cc:407] Connection header received
0620 02:23:34.893743 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:34.893750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:34.893838 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:34.893915 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:34.896805 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.897992 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.901313 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:34.901971 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:34.902121 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:34.907810 (+ 5689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:34.907833 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:34.907855 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:34.907907 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:34.912278 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:34.913127 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:34.913138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:34.913142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:34.913316 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:34.913968 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:34.913975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:34.913981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:34.914494 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:34.914855 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:34.915179 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 02:23:34.915443 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:23:34.916695 21765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:23:35.062608 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 21637
I20260620 02:23:35.081007 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 21776
Jun 20 02:23:35 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:23:35.097944 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:35.081867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.9.231.66:44045)
0620 02:23:35.082050 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:35.082056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:35.082076 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:23:35.082294 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:35.082309 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:35.082373 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:35.082492 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:35.083925 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:35.084762 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:35.086075 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:35.086503 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:35.086636 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:35.091216 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:35.091231 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:35.091248 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:35.091287 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:35.094490 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:35.095374 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:35.095386 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:35.095390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:35.095632 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:35.096079 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:35.096091 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:35.096094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:35.096584 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:35.096921 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:35.097286 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 02:23:35.097522 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:23:35.098729 21908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:23:35.217701 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 21776
I20260620 02:23:35.236730 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 21919
Jun 20 02:23:35 dist-test-slave-9vg1 krb5kdc[21081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:23:35.253865 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:23:35.237680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.9.231.67:43497)
0620 02:23:35.237842 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:23:35.237847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:23:35.237866 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:23:35.237988 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:23:35.237992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:23:35.238060 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:23:35.238163 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:23:35.239834 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:35.240592 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:35.242037 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:23:35.242507 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:23:35.242614 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:23:35.247712 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:23:35.247729 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:23:35.247746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:23:35.247790 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:23:35.250636 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:35.251616 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:35.251624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:35.251627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:35.251764 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:23:35.252205 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:23:35.252209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:23:35.252211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:23:35.252611 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:23:35.252917 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 02:23:35.253264 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 02:23:35.253489 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 02:23:35.254618 22042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:23:35.350368 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 21919
I20260620 02:23:35.368212 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 21525
I20260620 02:23:35.369830 21587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:23:35.487257 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 21525
I20260620 02:23:35.511693 10141 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:23:35.847985 10141 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:23:35.848268 10141 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:23:36.194607 10141 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:23:36.194 UTC [21111] LOG: received smart shutdown request
2026-06-20 02:23:36.198 UTC [21111] LOG: background worker "logical replication launcher" (PID 21120) exited with exit code 1
2026-06-20 02:23:36.199 UTC [21115] LOG: shutting down
2026-06-20 02:23:36.390 UTC [21111] LOG: database system is shut down
2026-06-20T02:23:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54211 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:23:36 dist-test-slave-9vg1 krb5kdc[22106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:23:36 dist-test-slave-9vg1 krb5kdc[22106](info): set up 2 sockets
Jun 20 02:23:36 dist-test-slave-9vg1 krb5kdc[22106](info): commencing operation
krb5kdc: starting...
W20260620 02:23:37.664525 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:23:37 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:23:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/rangeradmin_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/rangeradmin_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/rangerlookup_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/rangerlookup_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
I20260620 02:23:38.010125 10141 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4D_H_y/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/postgres -l logfile start
2026-06-20 02:23:40.682 UTC [22136] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:23:40.682 UTC [22136] LOG: listening on IPv4 address "127.9.231.84", port 58801
2026-06-20 02:23:40.686 UTC [22136] LOG: listening on Unix socket "/tmp/.s.PGSQL.58801"
2026-06-20 02:23:40.710 UTC [22139] LOG: database system was shut down at 2026-06-20 02:23:39 UTC
2026-06-20 02:23:40.714 UTC [22136] LOG: database system is ready to accept connections
I20260620 02:23:41.704217 10141 mini_postgres.cc:96] Postgres bound to 58801
2026-06-20 02:23:41.723 UTC [22147] FATAL: database "slave" does not exist
127.9.231.84:58801 - accepting connections
I20260620 02:23:41.724596 10141 mini_ranger.cc:161] Starting Ranger...
I20260620 02:23:41.752609 10141 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:23:42.296767 10141 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:23:42.296895 10141 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-admin
2026-06-20 02:23:42,596 [I] DB FLAVOR :POSTGRES
2026-06-20 02:23:42,597 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:23:42,597 [I] Checking connection..
2026-06-20 02:23:42,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:23:42,866 [I] Checking connection passed.
2026-06-20 02:23:42,867 [I] --------- Verifying version history table ---------
2026-06-20 02:23:42,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:23:43,146 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:23:43,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:23:43,385 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:23:43,385 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:23:43,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:23:43.658 UTC [22211] WARNING: there is no transaction in progress
2026-06-20 02:23:43,665 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:23:43,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:23:43,929 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:23:43,929 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:23:43,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:23:44,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:23:44,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9vg1.c.gcp-upstream.internal','N') ;"
2026-06-20 02:23:44,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:23:44,968 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:23:44,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:23:45,239 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:23:45,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:23:45,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:23:45,754 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:23:45,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:23:46.765 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.783 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.800 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.817 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.833 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.851 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.926 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.934 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.948 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.959 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.970 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.985 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:46.996 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:47.003 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:47.011 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:47.018 UTC [22319] WARNING: there is no transaction in progress
2026-06-20 02:23:47,705 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:23:47,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:23:47,958 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:23:47,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:23:48,203 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:23:48,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:23:48,485 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:23:48,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:23:48,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:23:49,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9vg1.c.gcp-upstream.internal';"
2026-06-20 02:23:49,257 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:23:49,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:23:49,499 [I] DB_PATCHES have already been applied
I20260620 02:23:49.505331 10141 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-admin:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4D_H_y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:23:49.505424 10141 mini_ranger.cc:193] Using host: 127.9.231.84
I20260620 02:23:49.520524 10141 mini_ranger.cc:227] Ranger admin URL: http://127.9.231.84:42167
Jun 20, 2026 2:23:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:23:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:23:50.670712 10141 mini_ranger.cc:160] Time spent starting Ranger: real 8.946s user 0.001s sys 0.040s
Jun 20, 2026 2:24:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:24:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:24:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:24:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:24:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:24:13.890019 10141 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.231.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.231.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/rangerkms_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.231.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/rangerkms_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.231.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.231.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.231.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/HTTP_127.9.231.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:24:14 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922254, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:24:14.082562 10141 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:24:14.104938 10141 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:24:15.094010 10141 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:24:15.094126 10141 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms
I20260620 02:24:15.094173 10141 mini_ranger_kms.cc:227] Using postgres at 127.9.231.84:58801
2026-06-20 02:24:15,468 [I] DB FLAVOR :POSTGRES
2026-06-20 02:24:15,468 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:24:15,468 [I] Checking connection
2026-06-20 02:24:15,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:24:15,722 [I] connection success
2026-06-20 02:24:15,723 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:24:15,723 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:24:15,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:24:15,980 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:24:15,981 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:24:15,981 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:24:15,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.231.84:58801/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:24:16,243 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:24:16.784648 10141 mini_ranger_kms.cc:310] Created kms service
I20260620 02:24:17.009605 10141 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:24:17.059496 10141 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:24:17 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922254, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:17.486030 10141 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:24:17.486393 10141 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms:/tmp/dist-test-task4D_H_y/build/asan/bin/postgresql.jar:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4D_H_y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:24:17.486461 10141 mini_ranger_kms.cc:240] Using host: 127.9.231.84
I20260620 02:24:17.505622 10141 mini_ranger_kms.cc:276] Ranger KMS PID: 22500
I20260620 02:24:17.505781 10141 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.231.84:55535
Jun 20, 2026 2:24:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:24:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:24:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:24:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:24:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:24:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 02:24:19 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922259, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.231.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:19.778951 10141 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.696s user 0.076s sys 0.316s
I20260620 02:24:19.779268 10141 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:24:19.779470 10141 mini_ranger_kms.cc:401] 127.9.231.84:55535/kms/v1/keys
Jun 20 02:24:20 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922259, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.231.84@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
Jun 20 02:24:20 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922259, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.231.84@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
Jun 20 02:24:22 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:24:23.027357 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34459
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34459
--ranger_config_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:23.381819 22550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:23.382143 22550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:23.382289 22550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:23.389892 22550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:23.390021 22550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:23.390115 22550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:23.390220 22550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:23.390288 22550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:23.400910 22550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.231.84:55535/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34459
--ranger_config_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:23.402312 22550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:23.403733 22550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:23.416543 22558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:23.416661 22555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:23.416667 22556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:23.417543 22550 server_base.cc:1034] running on GCE node
I20260620 02:24:23.418390 22550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:23.419986 22550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:23.421224 22550 hybrid_clock.cc:648] HybridClock initialized: now 1781922263421189 us; error 51 us; skew 500 ppm
I20260620 02:24:23.421618 22550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:23 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:23.425431 22550 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:24:23.427029 22550 webserver.cc:460] Webserver started at http://127.9.231.126:40949/ using document root <none> and password file <none>
I20260620 02:24:23.427800 22550 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:23.427956 22550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:23.428288 22550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:24:23 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:23.478953 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "cdc7a1e781b24c1684b8aee3fe467dba"
format_stamp: "Formatted at 2026-06-20 02:24:23 on dist-test-slave-9vg1"
server_key: "65b50f4816fb0104ae22e7c5321d01f9"
server_key_iv: "b65606d9c65ab2eff605e25c39d74e8f"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:23.479992 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "cdc7a1e781b24c1684b8aee3fe467dba"
format_stamp: "Formatted at 2026-06-20 02:24:23 on dist-test-slave-9vg1"
server_key: "65b50f4816fb0104ae22e7c5321d01f9"
server_key_iv: "b65606d9c65ab2eff605e25c39d74e8f"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:23.485918 22550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:24:23.502825 22565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:23.504550 22550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:24:23.504791 22550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "cdc7a1e781b24c1684b8aee3fe467dba"
format_stamp: "Formatted at 2026-06-20 02:24:23 on dist-test-slave-9vg1"
server_key: "65b50f4816fb0104ae22e7c5321d01f9"
server_key_iv: "b65606d9c65ab2eff605e25c39d74e8f"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:23.505043 22550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:23.522905 22550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:23.527738 22550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:23.528102 22550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:23.549021 22550 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34459
I20260620 02:24:23.549041 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34459 every 8 connection(s)
I20260620 02:24:23.550485 22550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:24:23.554589 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:23.557564 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 22550
I20260620 02:24:23.557781 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:24:23.558616 10141 external_mini_cluster.cc:1428] Setting key 4f9f25623cd12b2e8408cdef18372bd3
I20260620 02:24:23.568184 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Bootstrap starting.
I20260620 02:24:23.572331 22617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:23.573662 22617 log.cc:824] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:24:23 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:23.576797 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: No bootstrap required, opened a new log
I20260620 02:24:23.582818 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:23.560889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.9.231.126:34459)
0620 02:24:23.561754 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:23.561766 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:23.561808 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:24:23.563563 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:23.563586 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:23.563982 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:23.564320 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:23.565966 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:23.567367 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:23.569203 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:23.569969 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:23.570116 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:23.575164 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:23.575174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:23.575191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:23.575230 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:23.578347 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:23.579180 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:23.579189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:23.579205 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:23.579376 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:23.579785 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:23.579791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:23.579800 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:23.580395 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:23.580728 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:23.581247 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 02:24:23.581689 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":392,"threads_started":1}
I20260620 02:24:23.585693 22617 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:23.586197 22617 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:23.586326 22617 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc7a1e781b24c1684b8aee3fe467dba, State: Initialized, Role: FOLLOWER
I20260620 02:24:23.586875 22617 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:23.587157 22617 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:24:23.587266 22617 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:24:23.587391 22617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:23.590111 22617 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:23.590595 22617 leader_election.cc:302] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc7a1e781b24c1684b8aee3fe467dba; no voters:
I20260620 02:24:23.591243 22617 leader_election.cc:288] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:24:23.591385 22622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:24:23.592144 22622 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 LEADER]: Becoming Leader. State: Replica: cdc7a1e781b24c1684b8aee3fe467dba, State: Running, Role: LEADER
I20260620 02:24:23.592722 22622 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:23.593501 22617 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:23.596649 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } } }
I20260620 02:24:23.596547 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc7a1e781b24c1684b8aee3fe467dba. Latest consensus state: current_term: 1 leader_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } } }
I20260620 02:24:23.597051 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:23.597071 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:23.599131 22630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:24:23.605309 22630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:24:23.606154 22617 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:24:23.625864 22630 catalog_manager.cc:1347] Generated new cluster ID: 8a9b871b18174a249fe8a6b957970c43
I20260620 02:24:23.626031 22630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:24:23.670693 22630 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:24:23.672878 22630 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:24:23.692154 22630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Generated new TSK 0
I20260620 02:24:23.693399 22630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:24:24.431644 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34459
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:24:24 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:24:24.901717 22662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:24.902153 22662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:24.902314 22662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:24.914426 22662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:24.914610 22662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:24.914819 22662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:24:24.931281 22662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.231.84:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:24.933156 22662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:24.935115 22662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:24.953591 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:24.953848 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:24.963762 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:24.963799 22662 server_base.cc:1034] running on GCE node
I20260620 02:24:24.964828 22662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:24.966171 22662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:24.967478 22662 hybrid_clock.cc:648] HybridClock initialized: now 1781922264967359 us; error 105 us; skew 500 ppm
I20260620 02:24:24.967873 22662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:24 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:24.972324 22662 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:24:24.974567 22662 webserver.cc:460] Webserver started at http://127.9.231.65:40043/ using document root <none> and password file <none>
I20260620 02:24:24.975708 22662 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:24.975905 22662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:24.976315 22662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:24:24 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:25.001116 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "7148293119fe4fcaab321d16e125c91b"
format_stamp: "Formatted at 2026-06-20 02:24:24 on dist-test-slave-9vg1"
server_key: "9c8e8c08b5a4f64871ad3cb09625b253"
server_key_iv: "957949e58cbbdf01a0c3e58cdfbd9600"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:25.002501 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "7148293119fe4fcaab321d16e125c91b"
format_stamp: "Formatted at 2026-06-20 02:24:24 on dist-test-slave-9vg1"
server_key: "9c8e8c08b5a4f64871ad3cb09625b253"
server_key_iv: "957949e58cbbdf01a0c3e58cdfbd9600"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:25.010942 22662 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.004s
I20260620 02:24:25.030515 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:25.032487 22662 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:24:25.032696 22662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "7148293119fe4fcaab321d16e125c91b"
format_stamp: "Formatted at 2026-06-20 02:24:24 on dist-test-slave-9vg1"
server_key: "9c8e8c08b5a4f64871ad3cb09625b253"
server_key_iv: "957949e58cbbdf01a0c3e58cdfbd9600"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:25.032967 22662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:25.067649 22662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:25.073606 22662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:25.074079 22662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:25.083130 22662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:25.085472 22662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:25.085647 22662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:24:25.085835 22662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:25.085955 22662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:25.186667 22662 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:41399
I20260620 02:24:25.189350 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 22662
I20260620 02:24:25.189559 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:24:25.190383 10141 external_mini_cluster.cc:1428] Setting key b6a4a6229f8edc625b87169abc0f9879
I20260620 02:24:25.186707 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:41399 every 8 connection(s)
I20260620 02:24:25.188735 22662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
Jun 20 02:24:25 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Jun 20 02:24:25 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:25.260141 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:25.212063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:39165 (local address 127.9.231.126:34459)
0620 02:24:25.230308 (+ 18245us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:25.230317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:25.230346 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:24:25.234291 (+ 3945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:25.234296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:25.234393 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:25.234564 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:25.240586 (+ 6022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:25.241899 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:25.245409 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:25.246143 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:25.246259 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:25.251613 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:25.251623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:25.251629 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:25.251694 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:25.255689 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:25.256814 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:25.256822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:25.256826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:25.257017 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:25.257634 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:25.257640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:25.257644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:25.258070 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:25.258215 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:25.259409 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0620 02:24:25.259672 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18098,"thread_start_us":18267,"threads_started":1}
I20260620 02:24:25.263039 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:25.212841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34459 (local address 127.9.231.65:39165)
0620 02:24:25.229084 (+ 16243us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:25.229149 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:25.231995 (+ 2846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:25.234880 (+ 2885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:25.234895 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:25.235623 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:25.240258 (+ 4635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:25.240299 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:25.242802 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:25.242810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:25.245163 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:25.245186 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:25.246696 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:25.246705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:25.247087 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:25.248018 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:25.248058 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:25.251243 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:25.255912 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:25.255924 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:25.255958 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:25.256635 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:25.257192 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:25.257197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:25.257202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:25.257485 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:25.258317 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:25.258327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:25.258858 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 02:24:25.260691 (+ 1833us) client_negotiation.cc:240] Negotiation successful
0620 02:24:25.261428 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15863,"thread_start_us":19444,"threads_started":1}
I20260620 02:24:25.265455 22795 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34459
I20260620 02:24:25.265836 22795 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:25.266832 22795 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
I20260620 02:24:25.269861 22582 ts_manager.cc:194] Registered new tserver with Master: 7148293119fe4fcaab321d16e125c91b (127.9.231.65:41399)
I20260620 02:24:25.272100 22582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:39165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:24:25.290686 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34459
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:24:25 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
W20260620 02:24:25.790732 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:25.791313 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:25.791519 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:25.804215 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:25.804491 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:25.804745 22801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:24:25.821594 22801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.231.84:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:25.823755 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:25.825712 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:25.859598 22809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:25.860111 22811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:25.870626 22801 server_base.cc:1034] running on GCE node
W20260620 02:24:25.871615 22813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:25.872460 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:25.873598 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:25.874863 22801 hybrid_clock.cc:648] HybridClock initialized: now 1781922265874793 us; error 65 us; skew 500 ppm
I20260620 02:24:25.875294 22801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:25 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922265, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:25.883304 22801 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:24:25.885231 22801 webserver.cc:460] Webserver started at http://127.9.231.66:39513/ using document root <none> and password file <none>
I20260620 02:24:25.885978 22801 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:25.886109 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:25.886449 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:24:25 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922265, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:25.903900 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "1d7882ba8ef74941bc8d41a08d829cd2"
format_stamp: "Formatted at 2026-06-20 02:24:25 on dist-test-slave-9vg1"
server_key: "1bcd794f11cfecd2a4930f2639528b1c"
server_key_iv: "a0ade5934f59ed63f58ade02e9217f4b"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:25.904809 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "1d7882ba8ef74941bc8d41a08d829cd2"
format_stamp: "Formatted at 2026-06-20 02:24:25 on dist-test-slave-9vg1"
server_key: "1bcd794f11cfecd2a4930f2639528b1c"
server_key_iv: "a0ade5934f59ed63f58ade02e9217f4b"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:25.910588 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:24:25.931761 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:25.933777 22801 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260620 02:24:25.934062 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "1d7882ba8ef74941bc8d41a08d829cd2"
format_stamp: "Formatted at 2026-06-20 02:24:25 on dist-test-slave-9vg1"
server_key: "1bcd794f11cfecd2a4930f2639528b1c"
server_key_iv: "a0ade5934f59ed63f58ade02e9217f4b"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:25.934417 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:25.966941 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:25.972581 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:25.973004 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:25.974318 22801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:25.976301 22801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:25.976467 22801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 02:24:25.976660 22801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:25.976761 22801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:26.047497 22801 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:42287
I20260620 02:24:26.047509 22937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:42287 every 8 connection(s)
I20260620 02:24:26.049314 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:24:26.051512 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 22801
I20260620 02:24:26.051687 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:24:26.052243 10141 external_mini_cluster.cc:1428] Setting key 31e753653be5c6f88eb9250c1378a136
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Jun 20 02:24:26 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922265, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:26.113225 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:26.066549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34459 (local address 127.9.231.66:58125)
0620 02:24:26.075950 (+ 9401us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:26.076023 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:26.083552 (+ 7529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:26.084495 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:26.084507 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:26.085173 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:26.085991 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:26.086018 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.091290 (+ 5272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.091296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:26.093412 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:26.093442 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.094739 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.094749 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:26.095088 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:26.095938 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:26.095971 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:26.098874 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:26.108246 (+ 9372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:26.108259 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:26.108281 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:26.108951 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:26.109430 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:26.109434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:26.109437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:26.109741 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:26.110414 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:26.110424 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:26.110895 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 02:24:26.111217 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 02:24:26.111796 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8937,"thread_start_us":16407,"threads_started":1}
I20260620 02:24:26.115235 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:26.065716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:58125 (local address 127.9.231.126:34459)
0620 02:24:26.079501 (+ 13785us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:26.079505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:26.079527 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:24:26.083940 (+ 4413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:26.083946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:26.084062 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:26.084204 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:26.086182 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.087136 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.093650 (+ 6514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.094448 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.094582 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:26.104153 (+ 9571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:26.104162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:26.104167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:26.104220 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:26.108019 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.109141 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.109147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.109150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.109300 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.109900 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.109905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.109908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.110265 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:26.114439 (+ 4174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:26.114609 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 02:24:26.114859 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13647,"thread_start_us":13497,"threads_started":1}
I20260620 02:24:26.116856 22938 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34459
I20260620 02:24:26.117190 22938 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:26.118055 22938 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
I20260620 02:24:26.119959 22582 ts_manager.cc:194] Registered new tserver with Master: 1d7882ba8ef74941bc8d41a08d829cd2 (127.9.231.66:42287)
I20260620 02:24:26.121130 22582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:58125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:24:26.130564 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34459
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:26.276252 22795 heartbeater.cc:503] Master 127.9.231.126:34459 was elected leader, sending a full tablet report...
W20260620 02:24:26.556057 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:26.556468 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:26.556660 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:26.565354 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:26.565552 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:26.565760 22944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:24:26.576594 22944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.231.84:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:34459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:26.578037 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:26.579485 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:26.593946 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:26.594412 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:26.594412 22952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:26.594923 22944 server_base.cc:1034] running on GCE node
I20260620 02:24:26.595664 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:26.596757 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:26.598001 22944 hybrid_clock.cc:648] HybridClock initialized: now 1781922266597938 us; error 43 us; skew 500 ppm
I20260620 02:24:26.598449 22944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:26 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922266, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:26.602023 22944 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:24:26.603870 22944 webserver.cc:460] Webserver started at http://127.9.231.67:42605/ using document root <none> and password file <none>
I20260620 02:24:26.604636 22944 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:26.604782 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:26.605142 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:24:26 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922266, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:26.621320 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "3133fd5413754d4684a7211f516f9066"
format_stamp: "Formatted at 2026-06-20 02:24:26 on dist-test-slave-9vg1"
server_key: "6cf6b80025492738373a51482fa117aa"
server_key_iv: "42e68ee18a940a56e7df682a931b38f4"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:26.622334 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "3133fd5413754d4684a7211f516f9066"
format_stamp: "Formatted at 2026-06-20 02:24:26 on dist-test-slave-9vg1"
server_key: "6cf6b80025492738373a51482fa117aa"
server_key_iv: "42e68ee18a940a56e7df682a931b38f4"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:26.628199 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 02:24:26.644367 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:26.646307 22944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:24:26.646524 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "3133fd5413754d4684a7211f516f9066"
format_stamp: "Formatted at 2026-06-20 02:24:26 on dist-test-slave-9vg1"
server_key: "6cf6b80025492738373a51482fa117aa"
server_key_iv: "42e68ee18a940a56e7df682a931b38f4"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:26.646818 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:26.677925 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:26.681725 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:26.681993 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:26.682997 22944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:26.684461 22944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:26.684535 22944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:26.684617 22944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:26.684666 22944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:26.720119 22944 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:40349
I20260620 02:24:26.720157 23071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:40349 every 8 connection(s)
I20260620 02:24:26.721509 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:24:26.723861 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 22944
I20260620 02:24:26.724061 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:24:26.724745 10141 external_mini_cluster.cc:1428] Setting key 46dc922a0f630d121d107b62058b3d80
Jun 20 02:24:26 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922266, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:26.743914 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:26.724390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:40179 (local address 127.9.231.126:34459)
0620 02:24:26.724988 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:26.724992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:26.726230 (+ 1238us) server_negotiation.cc:407] Connection header received
0620 02:24:26.728693 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:26.728699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:26.728813 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:26.728931 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:26.730478 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.731263 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.732899 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.733358 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.733503 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:26.737058 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:26.737064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:26.737067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:26.737104 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:26.740430 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.741541 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.741547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.741550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.741714 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.742173 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.742176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.742178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.742454 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:26.742584 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:26.743474 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 02:24:26.743653 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":288,"threads_started":1}
I20260620 02:24:26.744932 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:26.725052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34459 (local address 127.9.231.67:40179)
0620 02:24:26.726060 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:26.726121 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:26.728438 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:26.729182 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:26.729192 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:26.729619 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:26.730308 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:26.730328 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.731449 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.731453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:26.732745 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:26.732757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.733594 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.733602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:26.733850 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:26.734587 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:26.734610 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:26.736812 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:26.740592 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:26.740605 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:26.740626 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:26.741423 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:26.741828 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:26.741831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:26.741833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:26.742036 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:26.742579 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:26.742587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:26.742991 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 02:24:26.743389 (+ 398us) client_negotiation.cc:240] Negotiation successful
0620 02:24:26.743798 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":391,"threads_started":1}
I20260620 02:24:26.746387 23072 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34459
I20260620 02:24:26.746688 23072 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:26.747515 23072 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
I20260620 02:24:26.749100 22582 ts_manager.cc:194] Registered new tserver with Master: 3133fd5413754d4684a7211f516f9066 (127.9.231.67:40349)
I20260620 02:24:26.750032 22582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:40179
I20260620 02:24:26.753916 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:24:26.755788 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 22662
Jun 20 02:24:26 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:24:26.776172 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:26.756770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.9.231.65:41399)
0620 02:24:26.757511 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:26.757521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:26.757555 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:24:26.757755 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:26.757772 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:26.758006 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:26.758186 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:26.760189 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.761500 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.763598 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.764343 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.764502 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:26.769918 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:26.769932 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:26.769948 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:26.769988 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:26.773106 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.773865 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.773872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.773875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.774016 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.774424 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.774434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.774436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.774834 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:26.775206 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:26.775506 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 02:24:26.775796 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":329,"threads_started":1}
I20260620 02:24:26.777500 22790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:26.868239 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 22662
I20260620 02:24:26.884917 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 22801
Jun 20 02:24:26 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:24:26.901719 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:26.885923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.9.231.66:42287)
0620 02:24:26.886525 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:26.886532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:26.886558 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:24:26.886619 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:26.886625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:26.886852 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:26.886988 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:26.888277 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.889318 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.890617 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:26.891280 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:26.891379 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:26.896016 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:26.896031 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:26.896049 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:26.896098 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:26.898901 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.899836 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.899844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.899846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.899981 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:26.900349 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:26.900352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:26.900354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:26.900695 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:26.900929 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:26.901192 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 02:24:26.901395 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":333,"threads_started":1}
I20260620 02:24:26.903049 22933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:26.985661 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 22801
I20260620 02:24:27.006619 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 22944
Jun 20 02:24:27 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:24:27.024744 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:27.007948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.9.231.67:40349)
0620 02:24:27.008658 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:27.008668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:27.008699 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:24:27.008759 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:27.008765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:27.008996 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:27.009121 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:27.010696 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:27.011848 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:27.013127 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:27.013695 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:27.013796 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:27.018393 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:27.018408 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:27.018424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:27.018467 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:27.021568 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:27.022342 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:27.022350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:27.022353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:27.022527 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:27.022944 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:27.022948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:27.022950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:27.023445 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:27.023733 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:27.024248 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 02:24:27.024458 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":395,"threads_started":1}
I20260620 02:24:27.026132 23067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:27.111518 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 22944
I20260620 02:24:27.127615 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 22550
I20260620 02:24:27.129006 22612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:27.238037 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 22550
I20260620 02:24:27.263410 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34459
--webserver_interface=127.9.231.126
--webserver_port=40949
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34459
--ranger_config_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:27.593385 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:27.593684 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:27.593773 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:27.601105 23084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:27.601228 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:27.601274 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:27.601329 23084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:27.601377 23084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:27.611651 23084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34459
--ranger_config_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=40949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:27.612951 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:27.614239 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:27.626572 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:27.626606 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:27.627271 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:27.628325 23084 server_base.cc:1034] running on GCE node
I20260620 02:24:27.629184 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:27.630793 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:27.632014 23084 hybrid_clock.cc:648] HybridClock initialized: now 1781922267631926 us; error 74 us; skew 500 ppm
I20260620 02:24:27.632409 23084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:27 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922267, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:27.635852 23084 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:24:27.637625 23084 webserver.cc:460] Webserver started at http://127.9.231.126:40949/ using document root <none> and password file <none>
I20260620 02:24:27.638358 23084 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:27.638531 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:24:27 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922267, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:27.732475 23084 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:24:27.736064 23100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:27.737821 23084 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:24:27.738034 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "cdc7a1e781b24c1684b8aee3fe467dba"
format_stamp: "Formatted at 2026-06-20 02:24:23 on dist-test-slave-9vg1"
server_key: "65b50f4816fb0104ae22e7c5321d01f9"
server_key_iv: "b65606d9c65ab2eff605e25c39d74e8f"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:27.739025 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:27.769496 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:27.773777 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:27.774086 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:27.793123 23084 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34459
I20260620 02:24:27.793141 23151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34459 every 8 connection(s)
I20260620 02:24:27.794770 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:24:27.798861 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23084
I20260620 02:24:27.799767 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:41399
--local_ip_for_outbound_sockets=127.9.231.65
--tserver_master_addrs=127.9.231.126:34459
--webserver_port=40043
--webserver_interface=127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:27.801887 23152 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:24:27.805191 23152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Bootstrap starting.
I20260620 02:24:27.824210 23152 log.cc:824] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:27.833189 23152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:24:27.833747 23152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Bootstrap complete.
I20260620 02:24:27.843212 23152 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:27.844230 23152 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc7a1e781b24c1684b8aee3fe467dba, State: Initialized, Role: FOLLOWER
I20260620 02:24:27.844955 23152 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:27.845217 23152 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:24:27.845355 23152 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:24:27.845539 23152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:24:27.849619 23152 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:27.850257 23152 leader_election.cc:302] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc7a1e781b24c1684b8aee3fe467dba; no voters:
I20260620 02:24:27.851003 23152 leader_election.cc:288] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:24:27.851193 23156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:24:27.853834 23152 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:27.855882 23156 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [term 2 LEADER]: Becoming Leader. State: Replica: cdc7a1e781b24c1684b8aee3fe467dba, State: Running, Role: LEADER
I20260620 02:24:27.856590 23156 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } }
I20260620 02:24:27.859496 23157 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } } }
I20260620 02:24:27.860836 23157 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:27.861433 23158 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc7a1e781b24c1684b8aee3fe467dba. Latest consensus state: current_term: 2 leader_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7a1e781b24c1684b8aee3fe467dba" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34459 } } }
I20260620 02:24:27.861807 23158 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:27.861850 23162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:24:27.875826 23162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:24:27.877295 23162 catalog_manager.cc:1259] Loaded cluster ID: 8a9b871b18174a249fe8a6b957970c43
I20260620 02:24:27.877399 23162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:24:27.881458 23162 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:24:27.882922 23162 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Loaded TSK: 0
I20260620 02:24:27.884505 23162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:24:27.894872 23152 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 02:24:28.251729 23154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:28.252146 23154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:28.252316 23154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:28.261716 23154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:28.261868 23154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:28.262090 23154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:24:28.275470 23154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:41399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=40043
--tserver_master_addrs=127.9.231.126:34459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:28.277227 23154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:28.278942 23154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:28.317000 23190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:28.320299 23192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:28.322639 23189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:28.324173 23154 server_base.cc:1034] running on GCE node
I20260620 02:24:28.324927 23154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:28.327384 23154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:28.328629 23154 hybrid_clock.cc:648] HybridClock initialized: now 1781922268328557 us; error 61 us; skew 500 ppm
I20260620 02:24:28.329020 23154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:28 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922268, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:28.332597 23154 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:24:28.334985 23154 webserver.cc:460] Webserver started at http://127.9.231.65:40043/ using document root <none> and password file <none>
I20260620 02:24:28.335918 23154 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:28.336083 23154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:24:28 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922268, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:28.364603 23154 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:24:28.374305 23200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:28.376688 23154 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260620 02:24:28.376950 23154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "7148293119fe4fcaab321d16e125c91b"
format_stamp: "Formatted at 2026-06-20 02:24:24 on dist-test-slave-9vg1"
server_key: "9c8e8c08b5a4f64871ad3cb09625b253"
server_key_iv: "957949e58cbbdf01a0c3e58cdfbd9600"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:28.378612 23154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:28.422534 23154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:28.427963 23154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:28.428398 23154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:28.431836 23154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:28.437937 23154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:28.438076 23154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:28.438191 23154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:28.438264 23154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:28.505159 23154 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:41399
I20260620 02:24:28.505208 23313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:41399 every 8 connection(s)
I20260620 02:24:28.506652 23154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:24:28.508178 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23154
I20260620 02:24:28.509025 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:42287
--local_ip_for_outbound_sockets=127.9.231.66
--tserver_master_addrs=127.9.231.126:34459
--webserver_port=39513
--webserver_interface=127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:24:28 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922268, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:28.546938 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:28.510865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34459 (local address 127.9.231.65:35607)
0620 02:24:28.521062 (+ 10197us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:28.521129 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:28.523896 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:28.525903 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:28.525916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:28.526489 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:28.527038 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:28.527123 (+ 85us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:28.529611 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:28.529617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:28.531655 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:28.531673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:28.533577 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:28.533585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:28.533854 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:28.534739 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:28.534772 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:28.537715 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:28.541922 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:28.541935 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:28.541954 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:28.542532 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:28.543108 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:28.543113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:28.543117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:28.543418 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:28.544343 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:28.544351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:28.544788 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 02:24:28.545101 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 02:24:28.545697 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9847,"thread_start_us":11459,"threads_started":1}
I20260620 02:24:28.548529 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:28.511990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:35607 (local address 127.9.231.126:34459)
0620 02:24:28.519474 (+ 7484us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:28.519497 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:28.522460 (+ 2963us) server_negotiation.cc:407] Connection header received
0620 02:24:28.524675 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:28.524698 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:28.525242 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:28.525767 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:28.527349 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:28.529396 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:28.531856 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:28.532949 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:28.533175 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:28.538107 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:28.538120 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:28.538141 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:28.538193 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:28.541762 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:28.542697 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:28.542707 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:28.542714 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:28.542925 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:28.543572 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:28.543578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:28.543580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:28.544203 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:28.544402 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:28.547396 (+ 2994us) server_negotiation.cc:299] Negotiation successful
0620 02:24:28.547727 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7179,"thread_start_us":6912,"threads_started":1}
I20260620 02:24:28.552050 23314 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34459
I20260620 02:24:28.552386 23314 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:28.553112 23314 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
W20260620 02:24:28.556325 23314 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:34459 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:24:28.556553 23314 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:28.556819 23314 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
I20260620 02:24:28.557888 23314 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:28.558151 23314 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
W20260620 02:24:28.559176 23314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:24:28.964428 23318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:28.964874 23318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:28.965031 23318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:28.976795 23318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:28.976956 23318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:28.977138 23318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:24:28.993194 23318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:42287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=39513
--tserver_master_addrs=127.9.231.126:34459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:28.995023 23318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:28.996841 23318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:29.012404 23330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:29.012439 23331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:29.012998 23333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:29.014851 23318 server_base.cc:1034] running on GCE node
I20260620 02:24:29.015579 23318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:29.016741 23318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:29.018018 23318 hybrid_clock.cc:648] HybridClock initialized: now 1781922269017945 us; error 62 us; skew 500 ppm
I20260620 02:24:29.018399 23318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:29.021787 23318 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:24:29.028282 23318 webserver.cc:460] Webserver started at http://127.9.231.66:39513/ using document root <none> and password file <none>
I20260620 02:24:29.029326 23318 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:29.029500 23318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:29.056958 23318 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:24:29.060137 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:29.061658 23318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:24:29.061851 23318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "1d7882ba8ef74941bc8d41a08d829cd2"
format_stamp: "Formatted at 2026-06-20 02:24:25 on dist-test-slave-9vg1"
server_key: "1bcd794f11cfecd2a4930f2639528b1c"
server_key_iv: "a0ade5934f59ed63f58ade02e9217f4b"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:29.062924 23318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:29.085547 23318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:29.090056 23318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:29.090363 23318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:29.091651 23318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:29.093405 23318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:29.093520 23318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:29.093638 23318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:29.093735 23318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:29.191287 23318 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:42287
I20260620 02:24:29.191360 23457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:42287 every 8 connection(s)
I20260620 02:24:29.193295 23318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:24:29.199890 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23318
I20260620 02:24:29.200919 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.231.84:55535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:40349
--local_ip_for_outbound_sockets=127.9.231.67
--tserver_master_addrs=127.9.231.126:34459
--webserver_port=42605
--webserver_interface=127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:29.232249 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:29.196194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:37697 (local address 127.9.231.126:34459)
0620 02:24:29.208805 (+ 12611us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:29.208813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:29.209730 (+ 917us) server_negotiation.cc:407] Connection header received
0620 02:24:29.212552 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:29.212559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:29.212674 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:29.212762 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:29.215237 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.216456 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.219019 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.219636 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.219752 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:29.224525 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:29.224533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:29.224537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:29.224582 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:29.228170 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.229205 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.229212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.229215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.229373 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.229984 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.229991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.229997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.230379 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:29.230515 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:29.231620 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0620 02:24:29.231862 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12444,"thread_start_us":12624,"threads_started":1}
I20260620 02:24:29.235116 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:29.196812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34459 (local address 127.9.231.66:37697)
0620 02:24:29.209508 (+ 12696us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:29.209568 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:29.212180 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:29.213368 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:29.213381 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:29.214065 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:29.214979 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:29.215007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.216718 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.216724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:29.218823 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:29.218840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.220116 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.220125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:29.220419 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:29.221276 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:29.221319 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:29.224218 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:29.228365 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:29.228378 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:29.228399 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:29.229048 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:29.229529 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:29.229534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:29.229538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:29.229842 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:29.230622 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:29.230630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:29.231140 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 02:24:29.232874 (+ 1734us) client_negotiation.cc:240] Negotiation successful
0620 02:24:29.233504 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12364,"thread_start_us":12158,"threads_started":1}
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:29.237669 23458 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34459
I20260620 02:24:29.238040 23458 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:29.239012 23458 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
I20260620 02:24:29.241318 23117 ts_manager.cc:194] Registered new tserver with Master: 1d7882ba8ef74941bc8d41a08d829cd2 (127.9.231.66:42287)
I20260620 02:24:29.243775 23117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:37697
I20260620 02:24:29.559350 23314 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:29.559823 23314 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
I20260620 02:24:29.561287 23117 ts_manager.cc:194] Registered new tserver with Master: 7148293119fe4fcaab321d16e125c91b (127.9.231.65:41399)
I20260620 02:24:29.562536 23117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:35607
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
W20260620 02:24:29.714501 23462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:29.714911 23462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:29.715051 23462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:29.726431 23462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:29.726588 23462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:29.726822 23462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:24:29.742420 23462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.231.84:55535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:40349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=42605
--tserver_master_addrs=127.9.231.126:34459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:29.744349 23462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:29.746013 23462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:29.763945 23469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:29.764721 23468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:29.777557 23462 server_base.cc:1034] running on GCE node
W20260620 02:24:29.777839 23471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:29.778589 23462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:29.779835 23462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:29.781095 23462 hybrid_clock.cc:648] HybridClock initialized: now 1781922269781026 us; error 56 us; skew 500 ppm
I20260620 02:24:29.781519 23462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:29.785204 23462 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:24:29.788554 23462 webserver.cc:460] Webserver started at http://127.9.231.67:42605/ using document root <none> and password file <none>
I20260620 02:24:29.789310 23462 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:29.789479 23462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:29.816540 23462 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 02:24:29.819751 23479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:29.821267 23462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:24:29.821460 23462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "3133fd5413754d4684a7211f516f9066"
format_stamp: "Formatted at 2026-06-20 02:24:26 on dist-test-slave-9vg1"
server_key: "6cf6b80025492738373a51482fa117aa"
server_key_iv: "42e68ee18a940a56e7df682a931b38f4"
server_key_version: "kuduclusterkey@0"
I20260620 02:24:29.822486 23462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:29.844934 23462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:29.849361 23462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:29.849668 23462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:29.850841 23462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:29.852417 23462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:29.852545 23462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:29.852686 23462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:29.852774 23462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:29.902007 23462 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:40349
I20260620 02:24:29.902050 23591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:40349 every 8 connection(s)
I20260620 02:24:29.903833 23462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:24:29.911262 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23462
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:29.943724 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:29.909448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:49097 (local address 127.9.231.126:34459)
0620 02:24:29.919525 (+ 10077us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:29.919535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:29.919570 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:24:29.924240 (+ 4670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:29.924246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:29.924346 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:29.924437 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:29.926665 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.928012 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.930619 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.931358 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.931452 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:29.936461 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:29.936470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:29.936475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:29.936517 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:29.939952 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.940934 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.940939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.940942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.941101 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.941606 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.941615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.941618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.941982 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:29.942100 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:29.943095 (+ 995us) server_negotiation.cc:299] Negotiation successful
0620 02:24:29.943370 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9890,"thread_start_us":9719,"threads_started":1}
I20260620 02:24:29.946285 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:29.908790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:34459 (local address 127.9.231.67:49097)
0620 02:24:29.915765 (+ 6975us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:29.915828 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:29.923728 (+ 7900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:29.924940 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:29.924962 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:29.925557 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:29.926420 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:29.926445 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.928226 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.928232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:29.930435 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:29.930455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.931816 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.931823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:29.932136 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:29.932938 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:29.932986 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:29.936119 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:29.940154 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:29.940164 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:29.940185 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:29.940795 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:29.941251 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:29.941255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:29.941258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:29.941490 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:29.942205 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:29.942215 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:29.942657 (+ 442us) client_negotiation.cc:769] Sending connection context
0620 02:24:29.944279 (+ 1622us) client_negotiation.cc:240] Negotiation successful
0620 02:24:29.944867 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6590,"thread_start_us":6522,"threads_started":1}
I20260620 02:24:29.946478 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:29.912463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.9.231.126:34459)
0620 02:24:29.923843 (+ 11380us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:29.923849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:29.927243 (+ 3394us) server_negotiation.cc:407] Connection header received
0620 02:24:29.927433 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:29.927439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:29.927548 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:29.927653 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:29.930509 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.931806 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.933880 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.934573 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.934675 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:29.939189 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:29.939201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:29.939206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:29.939267 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:29.942867 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.943994 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.944000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.944003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.944139 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.944661 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.944666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.944670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.945018 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:29.945296 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:29.945870 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 02:24:29.946104 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11227,"thread_start_us":11054,"threads_started":1}
I20260620 02:24:29.948519 23592 heartbeater.cc:347] Connected to a master server at 127.9.231.126:34459
I20260620 02:24:29.948840 23592 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:29.949683 23592 heartbeater.cc:511] Master 127.9.231.126:34459 requested a full tablet report, sending...
I20260620 02:24:29.951303 23117 ts_manager.cc:194] Registered new tserver with Master: 3133fd5413754d4684a7211f516f9066 (127.9.231.67:40349)
I20260620 02:24:29.952463 23117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:49097
I20260620 02:24:29.963479 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:24:29 dist-test-slave-9vg1 krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922269, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for HTTP/127.9.231.84@KRBTEST.COM
I20260620 02:24:29.995944 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:29.976073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.9.231.126:34459)
0620 02:24:29.976292 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:29.976302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:29.976860 (+ 558us) server_negotiation.cc:407] Connection header received
0620 02:24:29.977176 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:29.977184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:29.977286 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:29.977367 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:29.979851 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.980965 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.983297 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:29.983943 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:29.984120 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:29.988269 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:29.988277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:29.988282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:29.988334 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:29.992310 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.993291 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.993298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.993302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.993468 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:29.994000 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:29.994006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:29.994009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:29.994404 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:29.994663 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:29.995339 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 02:24:29.995599 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:24:30.008080 23117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:24:30.020650 23117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:24:30.157644 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.138655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:41399 (local address 127.0.0.1:49668)
0620 02:24:30.146903 (+ 8248us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:30.146926 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:30.147836 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:30.149777 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:30.149782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:30.149813 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:30.150250 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.150264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.152533 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.152539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.154623 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.154641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.156834 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.156841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.157025 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.157059 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:24:30.157159 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 02:24:30.157294 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8096,"thread_start_us":8410,"threads_started":1}
I20260620 02:24:30.158124 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.105010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.9.231.66:42287)
0620 02:24:30.124210 (+ 19200us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.124220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.132377 (+ 8157us) server_negotiation.cc:407] Connection header received
0620 02:24:30.132724 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.132734 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.133027 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.133212 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:30.134135 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.137547 (+ 3412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.153845 (+ 16298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.155604 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.155737 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.155973 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.157456 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.157665 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19025,"thread_start_us":18890,"threads_started":1}
I20260620 02:24:30.159174 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.132679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.9.231.65:41399)
0620 02:24:30.148711 (+ 16032us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.148721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.148754 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:24:30.148864 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.148873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.149220 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.149449 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:30.150677 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.152343 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.154833 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.156308 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.156445 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.156668 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.158655 (+ 1987us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.158826 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15809,"thread_start_us":17721,"threads_started":1}
I20260620 02:24:30.157330 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.104737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:42287 (local address 127.0.0.1:41434)
0620 02:24:30.132236 (+ 27499us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:30.132258 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:30.132518 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:30.133547 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:30.133555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:30.133593 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:30.133956 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.133968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.137786 (+ 3818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.137792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.140292 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.140312 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.156200 (+ 15888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.156207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.156463 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.156521 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:24:30.156670 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 02:24:30.156909 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27342,"thread_start_us":27178,"threads_started":1}
I20260620 02:24:30.164083 23393 tablet_service.cc:1467] Processing CreateTablet for tablet 19f2292083144a9a9c7ad30972776366 (DEFAULT_TABLE table=test-table [id=3195161e630f41a2b99894be8c3aad29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:30.165268 23248 tablet_service.cc:1467] Processing CreateTablet for tablet 19f2292083144a9a9c7ad30972776366 (DEFAULT_TABLE table=test-table [id=3195161e630f41a2b99894be8c3aad29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:30.166132 23393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f2292083144a9a9c7ad30972776366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:30.166877 23248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f2292083144a9a9c7ad30972776366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:30.170485 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.102379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:40349 (local address 127.0.0.1:52884)
0620 02:24:30.116905 (+ 14526us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:30.116992 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:30.117266 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:30.150940 (+ 33674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:30.150946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:30.150978 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:30.151387 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.151400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.160115 (+ 8715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.160122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.162218 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.162237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.169433 (+ 7196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.169444 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.169684 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.169733 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 02:24:30.169865 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 02:24:30.170065 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13934,"thread_start_us":15169,"threads_started":1}
I20260620 02:24:30.176537 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.103232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.9.231.67:40349)
0620 02:24:30.140981 (+ 37749us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.140992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.141032 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:24:30.141151 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.141160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.141455 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.151626 (+ 10171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:30.151992 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.153623 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.167305 (+ 13682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.169102 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.169284 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.169527 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.175895 (+ 6368us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.176125 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37537,"thread_start_us":37363,"threads_started":1}
I20260620 02:24:30.180948 23527 tablet_service.cc:1467] Processing CreateTablet for tablet 19f2292083144a9a9c7ad30972776366 (DEFAULT_TABLE table=test-table [id=3195161e630f41a2b99894be8c3aad29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:30.182713 23527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f2292083144a9a9c7ad30972776366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:30.209360 23615 tablet_bootstrap.cc:492] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Bootstrap starting.
I20260620 02:24:30.215339 23615 tablet_bootstrap.cc:654] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:30.216813 23615 log.cc:824] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:30.224255 23615 tablet_bootstrap.cc:492] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: No bootstrap required, opened a new log
I20260620 02:24:30.224587 23615 ts_tablet_manager.cc:1397] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.007s
I20260620 02:24:30.231848 23617 tablet_bootstrap.cc:492] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Bootstrap starting.
I20260620 02:24:30.237550 23617 tablet_bootstrap.cc:654] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:30.237896 23615 raft_consensus.cc:348] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.238435 23615 raft_consensus.cc:374] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:30.238550 23615 raft_consensus.cc:729] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3133fd5413754d4684a7211f516f9066, State: Initialized, Role: FOLLOWER
I20260620 02:24:30.239532 23617 log.cc:824] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:30.247634 23615 consensus_queue.cc:260] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.247866 23619 tablet_bootstrap.cc:492] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Bootstrap starting.
I20260620 02:24:30.249285 23458 heartbeater.cc:503] Master 127.9.231.126:34459 was elected leader, sending a full tablet report...
I20260620 02:24:30.253970 23617 tablet_bootstrap.cc:492] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: No bootstrap required, opened a new log
I20260620 02:24:30.254324 23617 ts_tablet_manager.cc:1397] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.006s
I20260620 02:24:30.265304 23617 raft_consensus.cc:348] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.265946 23617 raft_consensus.cc:374] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:30.265969 23619 tablet_bootstrap.cc:654] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:30.266207 23617 raft_consensus.cc:729] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7882ba8ef74941bc8d41a08d829cd2, State: Initialized, Role: FOLLOWER
I20260620 02:24:30.267198 23617 consensus_queue.cc:260] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.267501 23619 log.cc:824] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:30.268523 23592 heartbeater.cc:503] Master 127.9.231.126:34459 was elected leader, sending a full tablet report...
I20260620 02:24:30.269217 23615 ts_tablet_manager.cc:1428] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Time spent starting tablet: real 0.044s user 0.019s sys 0.013s
I20260620 02:24:30.277175 23617 ts_tablet_manager.cc:1428] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
I20260620 02:24:30.280162 23619 tablet_bootstrap.cc:492] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: No bootstrap required, opened a new log
I20260620 02:24:30.280560 23619 ts_tablet_manager.cc:1397] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.004s
I20260620 02:24:30.288931 23619 raft_consensus.cc:348] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.289575 23619 raft_consensus.cc:374] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:30.289727 23619 raft_consensus.cc:729] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7148293119fe4fcaab321d16e125c91b, State: Initialized, Role: FOLLOWER
I20260620 02:24:30.290611 23619 consensus_queue.cc:260] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.298116 23314 heartbeater.cc:503] Master 127.9.231.126:34459 was elected leader, sending a full tablet report...
I20260620 02:24:30.298955 23619 ts_tablet_manager.cc:1428] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260620 02:24:30.404487 23623 raft_consensus.cc:482] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:24:30.404883 23623 raft_consensus.cc:504] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.406713 23623 leader_election.cc:288] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3133fd5413754d4684a7211f516f9066 (127.9.231.67:40349), 1d7882ba8ef74941bc8d41a08d829cd2 (127.9.231.66:42287)
I20260620 02:24:30.413669 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.407193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:59897 (local address 127.9.231.67:40349)
0620 02:24:30.407386 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.407394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.408017 (+ 623us) server_negotiation.cc:407] Connection header received
0620 02:24:30.408261 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.408269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.408346 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.408411 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:30.409253 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.410102 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.411979 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.412733 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.412921 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.412998 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.413315 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.413444 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 02:24:30.413671 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.407362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:53737 (local address 127.9.231.66:42287)
0620 02:24:30.407551 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.407556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.408153 (+ 597us) server_negotiation.cc:407] Connection header received
0620 02:24:30.408335 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.408338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.408429 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.408519 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:30.409123 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.409886 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.411904 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.412690 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.412805 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.412909 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.413252 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.413392 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:24:30.413763 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.407201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:42287 (local address 127.9.231.65:53737)
0620 02:24:30.408016 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:30.408036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:30.408191 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:30.408574 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:30.408578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:30.408610 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:30.408944 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.408954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.410017 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.410023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.411747 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.411759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.412906 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.412912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.413088 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.413112 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:24:30.413190 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 02:24:30.413335 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":82,"thread_start_us":518,"threads_started":1}
I20260620 02:24:30.413805 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.406933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:40349 (local address 127.9.231.65:59897)
0620 02:24:30.407872 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:30.407894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:30.408062 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:30.408464 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:30.408471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:30.408501 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:30.409013 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.409051 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.410253 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.410256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.411807 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:30.411843 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.413003 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.413008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:30.413135 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.413157 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:24:30.413272 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:24:30.413391 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"spinlock_wait_cycles":34688,"thread_start_us":494,"threads_started":1}
I20260620 02:24:30.414902 23547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2292083144a9a9c7ad30972776366" candidate_uuid: "7148293119fe4fcaab321d16e125c91b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3133fd5413754d4684a7211f516f9066" is_pre_election: true
I20260620 02:24:30.414902 23413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2292083144a9a9c7ad30972776366" candidate_uuid: "7148293119fe4fcaab321d16e125c91b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" is_pre_election: true
I20260620 02:24:30.415501 23547 raft_consensus.cc:2454] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7148293119fe4fcaab321d16e125c91b in term 0.
I20260620 02:24:30.415501 23413 raft_consensus.cc:2454] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7148293119fe4fcaab321d16e125c91b in term 0.
I20260620 02:24:30.416270 23201 leader_election.cc:302] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d7882ba8ef74941bc8d41a08d829cd2, 3133fd5413754d4684a7211f516f9066, 7148293119fe4fcaab321d16e125c91b; no voters:
I20260620 02:24:30.416736 23623 raft_consensus.cc:2781] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:24:30.416893 23623 raft_consensus.cc:482] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:24:30.417013 23623 raft_consensus.cc:3037] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:30.419356 23623 raft_consensus.cc:504] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.419945 23623 leader_election.cc:288] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [CANDIDATE]: Term 1 election: Requested vote from peers 3133fd5413754d4684a7211f516f9066 (127.9.231.67:40349), 1d7882ba8ef74941bc8d41a08d829cd2 (127.9.231.66:42287)
I20260620 02:24:30.420487 23413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2292083144a9a9c7ad30972776366" candidate_uuid: "7148293119fe4fcaab321d16e125c91b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7882ba8ef74941bc8d41a08d829cd2"
I20260620 02:24:30.420487 23547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2292083144a9a9c7ad30972776366" candidate_uuid: "7148293119fe4fcaab321d16e125c91b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3133fd5413754d4684a7211f516f9066"
I20260620 02:24:30.420717 23547 raft_consensus.cc:3037] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:30.420715 23413 raft_consensus.cc:3037] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:24:30.423810 23593 tablet.cc:2369] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:24:30.424034 23413 raft_consensus.cc:2454] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7148293119fe4fcaab321d16e125c91b in term 1.
I20260620 02:24:30.424039 23547 raft_consensus.cc:2454] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7148293119fe4fcaab321d16e125c91b in term 1.
I20260620 02:24:30.424463 23201 leader_election.cc:302] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7882ba8ef74941bc8d41a08d829cd2, 7148293119fe4fcaab321d16e125c91b; no voters:
I20260620 02:24:30.424747 23623 raft_consensus.cc:2781] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:24:30.425215 23623 raft_consensus.cc:686] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 1 LEADER]: Becoming Leader. State: Replica: 7148293119fe4fcaab321d16e125c91b, State: Running, Role: LEADER
I20260620 02:24:30.425675 23623 consensus_queue.cc:237] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } }
I20260620 02:24:30.430217 23117 catalog_manager.cc:5495] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7148293119fe4fcaab321d16e125c91b (127.9.231.65). New cstate: current_term: 1 leader_uuid: "7148293119fe4fcaab321d16e125c91b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7148293119fe4fcaab321d16e125c91b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 41399 } health_report { overall_health: HEALTHY } } }
W20260620 02:24:30.447388 23459 tablet.cc:2369] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:24:30.500034 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.494277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.9.231.65:41399)
0620 02:24:30.494518 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.494525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.494548 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:24:30.494816 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.494820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.494910 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.495067 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:24:30.495511 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.496277 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.497728 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.498137 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.498269 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.498804 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:24:30.499406 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:24:30.499567 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.499665 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.499748 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:24:30.508599 23547 raft_consensus.cc:1261] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 1 FOLLOWER]: Refusing update from remote peer 7148293119fe4fcaab321d16e125c91b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:30.508587 23413 raft_consensus.cc:1261] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 1 FOLLOWER]: Refusing update from remote peer 7148293119fe4fcaab321d16e125c91b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:30.509577 23623 consensus_queue.cc:1035] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3133fd5413754d4684a7211f516f9066" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 40349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:30.509938 23633 consensus_queue.cc:1035] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7882ba8ef74941bc8d41a08d829cd2" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:30.521274 23636 mvcc.cc:205] Tried to move back new op lower bound from 7298753619992510464 to 7298753619667607552. Current Snapshot: MvccSnapshot[applied={T|T < 7298753619992510464}]
I20260620 02:24:30.539566 23117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:24:30.539849 23117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:24:30.543395 23117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdc7a1e781b24c1684b8aee3fe467dba: Sending DeleteTablet for 3 replicas of tablet 19f2292083144a9a9c7ad30972776366
I20260620 02:24:30.544704 23527 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f2292083144a9a9c7ad30972776366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:30 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:52884
I20260620 02:24:30.544741 23248 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f2292083144a9a9c7ad30972776366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:30 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:49668
I20260620 02:24:30.545444 23393 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f2292083144a9a9c7ad30972776366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:30 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:41434
I20260620 02:24:30.545608 23646 tablet_replica.cc:330] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: stopping tablet replica
I20260620 02:24:30.546065 23646 raft_consensus.cc:2229] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:24:30.546638 23646 raft_consensus.cc:2258] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:30.547600 23647 tablet_replica.cc:330] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: stopping tablet replica
I20260620 02:24:30.549047 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23154
I20260620 02:24:30.550246 23646 ts_tablet_manager.cc:1905] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:30.556623 23648 tablet_replica.cc:330] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: stopping tablet replica
I20260620 02:24:30.557252 23648 raft_consensus.cc:2229] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:24:30.557868 23648 raft_consensus.cc:2258] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:30.559525 23646 ts_tablet_manager.cc:1918] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:30.559749 23646 log.cc:1196] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/19f2292083144a9a9c7ad30972776366
I20260620 02:24:30.559944 23648 ts_tablet_manager.cc:1905] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:30.560387 23646 ts_tablet_manager.cc:1939] T 19f2292083144a9a9c7ad30972776366 P 3133fd5413754d4684a7211f516f9066: Deleting consensus metadata
I20260620 02:24:30.562256 23104 catalog_manager.cc:4841] TS 3133fd5413754d4684a7211f516f9066 (127.9.231.67:40349): tablet 19f2292083144a9a9c7ad30972776366 (table test-table [id=3195161e630f41a2b99894be8c3aad29]) successfully deleted
I20260620 02:24:30.561949 23647 raft_consensus.cc:2229] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:24:30.563349 23647 raft_consensus.cc:2258] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:30.565747 23647 ts_tablet_manager.cc:1905] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:30.568885 23648 ts_tablet_manager.cc:1918] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:30.569149 23648 log.cc:1196] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/19f2292083144a9a9c7ad30972776366
I20260620 02:24:30.569795 23648 ts_tablet_manager.cc:1939] T 19f2292083144a9a9c7ad30972776366 P 1d7882ba8ef74941bc8d41a08d829cd2: Deleting consensus metadata
I20260620 02:24:30.571846 23101 catalog_manager.cc:4841] TS 1d7882ba8ef74941bc8d41a08d829cd2 (127.9.231.66:42287): tablet 19f2292083144a9a9c7ad30972776366 (table test-table [id=3195161e630f41a2b99894be8c3aad29]) successfully deleted
I20260620 02:24:30.573565 23647 ts_tablet_manager.cc:1918] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:30.573769 23647 log.cc:1196] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/19f2292083144a9a9c7ad30972776366
I20260620 02:24:30.574390 23647 ts_tablet_manager.cc:1939] T 19f2292083144a9a9c7ad30972776366 P 7148293119fe4fcaab321d16e125c91b: Deleting consensus metadata
I20260620 02:24:30.576341 23101 catalog_manager.cc:4841] TS 7148293119fe4fcaab321d16e125c91b (127.9.231.65:41399): tablet 19f2292083144a9a9c7ad30972776366 (table test-table [id=3195161e630f41a2b99894be8c3aad29]) successfully deleted
I20260620 02:24:30.578882 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.551582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.9.231.65:41399)
0620 02:24:30.552033 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.552040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.554580 (+ 2540us) server_negotiation.cc:407] Connection header received
0620 02:24:30.554772 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.554781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.562466 (+ 7685us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.562602 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:30.564644 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.565420 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.567704 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.568474 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.568614 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.572319 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:30.572332 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:30.572345 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:30.572390 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:30.575858 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:30.576742 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:30.576756 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:30.576758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:30.576891 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:30.577301 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:30.577305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:30.577307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:30.577670 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:30.577979 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.578318 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.578558 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260620 02:24:30.579571 23309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:30.678563 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23154
I20260620 02:24:30.698417 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23318
I20260620 02:24:30.712524 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.699377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.9.231.66:42287)
0620 02:24:30.699599 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.699606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.699760 (+ 154us) server_negotiation.cc:407] Connection header received
0620 02:24:30.699972 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.699981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.700045 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.700170 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:30.701461 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.702189 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.703725 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.704106 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.704256 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.706799 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:30.706812 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:30.706827 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:30.706857 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:30.709663 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:30.710375 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:30.710382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:30.710385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:30.710509 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:30.710896 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:30.710902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:30.710905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:30.711332 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:30.711631 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.711957 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.712182 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:24:30.713192 23453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:30.804122 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23318
I20260620 02:24:30.822576 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23462
I20260620 02:24:30.836143 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:30.823418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.9.231.67:40349)
0620 02:24:30.823589 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:30.823594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:30.823611 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:24:30.823690 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:30.823695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:30.823755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:30.823839 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:30.825175 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.826179 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.827554 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:30.827968 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:30.828065 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:30.830734 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:30.830750 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:30.830767 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:30.830816 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:30.833653 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:30.834320 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:30.834327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:30.834329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:30.834455 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:30.834777 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:30.834780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:30.834782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:30.835269 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:30.835489 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:30.835722 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 02:24:30.835877 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:24:30.836815 23587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:30.934561 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23462
I20260620 02:24:30.952049 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23084
I20260620 02:24:30.953307 23147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:31.067946 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23084
I20260620 02:24:31.090157 10141 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:24:31.429550 10141 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:24:31.429741 10141 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:24:31.775309 10141 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:24:31.775 UTC [22136] LOG: received smart shutdown request
2026-06-20 02:24:31.779 UTC [22136] LOG: background worker "logical replication launcher" (PID 22145) exited with exit code 1
2026-06-20 02:24:31.779 UTC [22140] LOG: shutting down
2026-06-20 02:24:31.969 UTC [22136] LOG: database system is shut down
2026-06-20T02:24:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55584 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:24:32 dist-test-slave-9vg1 krb5kdc[23662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:24:32 dist-test-slave-9vg1 krb5kdc[23662](info): set up 2 sockets
Jun 20 02:24:32 dist-test-slave-9vg1 krb5kdc[23662](info): commencing operation
krb5kdc: starting...
W20260620 02:24:33.349041 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:24:33 dist-test-slave-9vg1 krb5kdc[23662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:24:33.598547 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40695
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:40695,127.9.231.125:42775,127.9.231.124:36755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:33.946415 23675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:33.946755 23675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:33.946848 23675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:33.954419 23675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:33.954545 23675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:33.954597 23675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:33.954653 23675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:33.954704 23675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:33.964993 23675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:40695,127.9.231.125:42775,127.9.231.124:36755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:33.966370 23675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:33.967772 23675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:33.980682 23683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:33.980760 23681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:33.980767 23680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:33.981798 23675 server_base.cc:1034] running on GCE node
I20260620 02:24:33.982733 23675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:33.984485 23675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:33.985744 23675 hybrid_clock.cc:648] HybridClock initialized: now 1781922273985698 us; error 87 us; skew 500 ppm
I20260620 02:24:33.986136 23675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:33 dist-test-slave-9vg1 krb5kdc[23662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:33.994369 23675 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:24:33.996193 23675 webserver.cc:460] Webserver started at http://127.9.231.126:45131/ using document root <none> and password file <none>
I20260620 02:24:33.997001 23675 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:33.997159 23675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:33.997478 23675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:34.000758 23675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "b25692c655ab47338880f1dc3754a23b"
format_stamp: "Formatted at 2026-06-20 02:24:33 on dist-test-slave-9vg1"
server_key: "b54675cae51cf0a4bf25e3ed49143b2d"
server_key_iv: "7cafcbf2a111b26f78a6d270894baf30"
server_key_version: "encryptionkey@0"
I20260620 02:24:34.001637 23675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "b25692c655ab47338880f1dc3754a23b"
format_stamp: "Formatted at 2026-06-20 02:24:33 on dist-test-slave-9vg1"
server_key: "b54675cae51cf0a4bf25e3ed49143b2d"
server_key_iv: "7cafcbf2a111b26f78a6d270894baf30"
server_key_version: "encryptionkey@0"
I20260620 02:24:34.007445 23675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:24:34.011291 23690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:34.012876 23675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:24:34.013100 23675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "b25692c655ab47338880f1dc3754a23b"
format_stamp: "Formatted at 2026-06-20 02:24:33 on dist-test-slave-9vg1"
server_key: "b54675cae51cf0a4bf25e3ed49143b2d"
server_key_iv: "7cafcbf2a111b26f78a6d270894baf30"
server_key_version: "encryptionkey@0"
I20260620 02:24:34.013332 23675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:34.050972 23675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:34.055444 23675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:34.055751 23675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:34.075564 23675 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:40695
I20260620 02:24:34.075646 23741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:40695 every 8 connection(s)
I20260620 02:24:34.077062 23675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:24:34.081001 23742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:34.083662 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23675
I20260620 02:24:34.083912 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:24:34.084722 10141 external_mini_cluster.cc:1428] Setting key 9f6c5fe0cf36da8e950fc9c7633e1107
I20260620 02:24:34.091259 23742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.231.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.125@KRBTEST.COM" created.
Jun 20 02:24:34 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:34.125743 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.095653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.9.231.126:40695)
0620 02:24:34.097881 (+ 2228us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:34.097887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:34.097931 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:24:34.100353 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:34.100359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:34.100674 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:34.100853 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:34.102621 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.104027 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.106458 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.108204 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.108409 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.116677 (+ 8268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:34.116686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:34.116700 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:34.116737 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:34.120336 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.121360 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.121372 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.121378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.121757 (+ 379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.122332 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.122340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.122347 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.123246 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:34.123451 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:34.124222 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 02:24:34.124536 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2028,"spinlock_wait_cycles":81152,"thread_start_us":1736,"threads_started":1}
I20260620 02:24:34.125785 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.095653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40695 (local address 127.0.0.1:54196)
0620 02:24:34.096764 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.096819 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:34.099900 (+ 3081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:34.100945 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:34.100954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:34.101831 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:34.102355 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.102385 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.104214 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.104221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.106273 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.106296 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.108805 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.108818 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.109300 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.110150 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:34.110191 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:34.116373 (+ 6182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:34.120502 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.120525 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.120542 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.121182 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.121919 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.121926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.121930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.122185 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.123421 (+ 1236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:34.123430 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:34.123744 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 02:24:34.124057 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 02:24:34.124729 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":62,"thread_start_us":515,"threads_started":1}
I20260620 02:24:34.129969 23742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:24:34.131345 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.130764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58498)
0620 02:24:34.130968 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.131193 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 02:24:34.131765 23694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
I20260620 02:24:34.133646 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.133126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58512)
0620 02:24:34.133324 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.133423 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:24:34.134004 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:42775: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab.
Entry for principal kudu/127.9.231.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab.
I20260620 02:24:34.159329 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:42775
--webserver_interface=127.9.231.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab
--principal=kudu/127.9.231.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:40695,127.9.231.125:42775,127.9.231.124:36755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:34.183401 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } attempt: 1
I20260620 02:24:34.184731 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.184163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58518)
0620 02:24:34.184393 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.184534 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:24:34.185794 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.185402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58520)
0620 02:24:34.185553 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.185636 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 02:24:34.185974 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:42775: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
I20260620 02:24:34.254262 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } attempt: 2
I20260620 02:24:34.255734 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.255050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58526)
0620 02:24:34.255403 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.255528 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 02:24:34.256956 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.256492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58536)
0620 02:24:34.256693 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.256794 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:24:34.257174 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:42775: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
I20260620 02:24:34.348472 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } attempt: 3
I20260620 02:24:34.349786 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.349257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58538)
0620 02:24:34.349461 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.349647 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 02:24:34.350927 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.350575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58546)
0620 02:24:34.350717 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.350788 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:24:34.351263 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:42775: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
I20260620 02:24:34.494575 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } attempt: 4
I20260620 02:24:34.495941 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.495409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58556)
0620 02:24:34.495724 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.495805 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 02:24:34.497104 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.496738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58570)
0620 02:24:34.496883 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.496977 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:24:34.497336 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:42775: Network error: Client connection negotiation failed: client connection to 127.9.231.125:42775: connect: Connection refused (error 111)
W20260620 02:24:34.512113 23749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:34.512485 23749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:34.512708 23749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:34.520820 23749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:34.520985 23749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:34.521099 23749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:34.521204 23749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:34.521281 23749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:34.532344 23749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:40695,127.9.231.125:42775,127.9.231.124:36755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab
--principal=kudu/127.9.231.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.231.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:34.533967 23749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:34.535429 23749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:34.547768 23754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:34.547803 23755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:34.547979 23757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:34.548668 23749 server_base.cc:1034] running on GCE node
I20260620 02:24:34.549504 23749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:34.550993 23749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:34.552259 23749 hybrid_clock.cc:648] HybridClock initialized: now 1781922274552216 us; error 29 us; skew 500 ppm
I20260620 02:24:34.552666 23749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:34 dist-test-slave-9vg1 krb5kdc[23662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:34.556147 23749 init.cc:377] Logged in from keytab as kudu/127.9.231.125@KRBTEST.COM (short username kudu)
I20260620 02:24:34.557580 23749 webserver.cc:460] Webserver started at http://127.9.231.125:32929/ using document root <none> and password file <none>
I20260620 02:24:34.558240 23749 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:34.558338 23749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:34.558595 23749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:34.561642 23749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data/instance:
uuid: "de6518f92b694cbb818fec2bfb9a7bac"
format_stamp: "Formatted at 2026-06-20 02:24:34 on dist-test-slave-9vg1"
server_key: "13b8b9576df3e040dbb1ffcdac55d166"
server_key_iv: "796c668a9685a70a38e4e800afac691b"
server_key_version: "encryptionkey@0"
I20260620 02:24:34.562464 23749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/wal/instance:
uuid: "de6518f92b694cbb818fec2bfb9a7bac"
format_stamp: "Formatted at 2026-06-20 02:24:34 on dist-test-slave-9vg1"
server_key: "13b8b9576df3e040dbb1ffcdac55d166"
server_key_iv: "796c668a9685a70a38e4e800afac691b"
server_key_version: "encryptionkey@0"
I20260620 02:24:34.568033 23749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:24:34.571616 23764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:34.573028 23749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:24:34.573230 23749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/wal
uuid: "de6518f92b694cbb818fec2bfb9a7bac"
format_stamp: "Formatted at 2026-06-20 02:24:34 on dist-test-slave-9vg1"
server_key: "13b8b9576df3e040dbb1ffcdac55d166"
server_key_iv: "796c668a9685a70a38e4e800afac691b"
server_key_version: "encryptionkey@0"
I20260620 02:24:34.573437 23749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:34.603669 23749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:34.608106 23749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:34.608412 23749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:34.628109 23749 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.125:42775
I20260620 02:24:34.628140 23815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.125:42775 every 8 connection(s)
I20260620 02:24:34.629611 23749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
I20260620 02:24:34.633664 23816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:34.635574 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23749
I20260620 02:24:34.635793 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-1/wal/instance
I20260620 02:24:34.636266 10141 external_mini_cluster.cc:1428] Setting key 3992937d47d9ca6af19bd5e7867ffb4c
I20260620 02:24:34.640707 23816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:24:34 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.231.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.124@KRBTEST.COM" created.
I20260620 02:24:34.665699 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.643482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.9.231.126:40695)
0620 02:24:34.644010 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:34.644017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:34.645259 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 02:24:34.648354 (+ 3095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:34.648359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:34.648438 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:34.648560 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:34.650778 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.651636 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.654199 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.654651 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.654813 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.659008 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:34.659013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:34.659017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:34.659132 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:34.661967 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.662898 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.662903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.662907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.663039 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.663700 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.663703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.663705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.664001 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:34.664110 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:34.665204 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0620 02:24:34.665388 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
I20260620 02:24:34.667502 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.644094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40695 (local address 127.0.0.1:54200)
0620 02:24:34.645045 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.645097 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:34.647983 (+ 2886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:34.648843 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:34.648858 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:34.649711 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:34.650558 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.650586 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.651814 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.651822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.654027 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.654046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.654913 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.654920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.655228 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.656045 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:34.656071 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:34.658745 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:34.662146 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.662178 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.662201 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.662755 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.663247 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.663256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.663264 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.663551 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.664163 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:34.664176 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:34.664740 (+ 564us) client_negotiation.cc:769] Sending connection context
0620 02:24:34.665119 (+ 379us) client_negotiation.cc:240] Negotiation successful
0620 02:24:34.665787 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":443,"threads_started":1}
I20260620 02:24:34.669524 23816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:24:34 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:24:34.685752 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.670569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.9.231.125:42775)
0620 02:24:34.671194 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:34.671200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:34.671227 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:24:34.671274 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:34.671280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:34.671464 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:34.671626 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:34.672398 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.673477 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.674606 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.675273 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.675407 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.679401 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:34.679414 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:34.679428 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:34.679459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:34.682361 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.683392 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.683399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.683401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.683514 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.684035 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.684038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.684040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.684394 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:34.684558 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:34.685057 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 02:24:34.685415 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":194,"server-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260620 02:24:34.685874 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.670301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58574)
0620 02:24:34.670466 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.670481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:34.670572 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:34.671609 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:34.671614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:34.672095 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:34.672243 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.672251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.673616 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.673623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.674468 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.674478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.675418 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.675423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.675669 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.676299 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:34.676318 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:34.679178 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:34.682516 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.682525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.682544 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.683252 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.683641 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.683648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.683654 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.683899 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.684555 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:34.684563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:34.684743 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 02:24:34.684937 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 02:24:34.685301 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":7}
I20260620 02:24:34.688537 23816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:24:34.689864 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.689336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37074)
0620 02:24:34.689588 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.689699 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 02:24:34.690214 23766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
I20260620 02:24:34.691844 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.691519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37078)
0620 02:24:34.691658 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.691740 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 02:24:34.692077 23816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab.
Entry for principal kudu/127.9.231.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab.
I20260620 02:24:34.696751 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:36755
--webserver_interface=127.9.231.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab
--principal=kudu/127.9.231.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:40695,127.9.231.125:42775,127.9.231.124:36755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:34.741446 23816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 1
I20260620 02:24:34.742862 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.742215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37084)
0620 02:24:34.742455 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.742633 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:24:34.744042 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.743625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37090)
0620 02:24:34.743802 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.743896 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:24:34.744300 23816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
I20260620 02:24:34.796643 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } attempt: 5
Jun 20 02:24:34 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:24:34.811041 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.797689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.9.231.125:42775)
0620 02:24:34.797868 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:34.797875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:34.797897 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:24:34.798096 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:34.798101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:34.798170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:34.798281 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:34.799147 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.799863 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.801073 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.801530 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.801666 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.805438 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:34.805445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:34.805448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:34.805486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:34.808296 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.809164 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.809169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.809171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.809297 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:34.809739 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:34.809743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:34.809745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:34.810058 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:34.810174 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:34.810617 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 02:24:34.810796 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:24:34.811233 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.797500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58578)
0620 02:24:34.797724 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.797748 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:34.797896 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:34.798319 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:34.798324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:34.798745 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:34.798926 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.798936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.800014 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.800021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.800916 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:34.800930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:34.801686 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:34.801692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:34.801884 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:34.802550 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:34.802570 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:34.805195 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:34.808464 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.808470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.808474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.809031 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.809426 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:34.809431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:34.809434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:34.809607 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:34.810181 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:34.810186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:34.810372 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 02:24:34.810529 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 02:24:34.810855 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 02:24:34.812340 23742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:24:34.812520 23816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 2
I20260620 02:24:34.813480 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.813014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37092)
0620 02:24:34.813211 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.813324 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 02:24:34.813611 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.813179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37106)
0620 02:24:34.813350 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.813440 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 02:24:34.814625 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.814214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37122)
0620 02:24:34.814387 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.814471 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 02:24:34.814643 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.814289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37138)
0620 02:24:34.814489 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.814549 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 02:24:34.814890 23816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
W20260620 02:24:34.814884 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
I20260620 02:24:34.866123 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 1
I20260620 02:24:34.867383 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.866846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37140)
0620 02:24:34.867045 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.867213 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:24:34.868593 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.868123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37150)
0620 02:24:34.868390 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.868478 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260620 02:24:34.868821 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
I20260620 02:24:34.906136 23816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 3
I20260620 02:24:34.907586 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.907086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37160)
0620 02:24:34.907317 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.907415 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 02:24:34.908735 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.908295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37166)
0620 02:24:34.908530 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.908611 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 02:24:34.908931 23816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
I20260620 02:24:34.937047 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 2
I20260620 02:24:34.938375 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.937847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37176)
0620 02:24:34.938093 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.938212 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 02:24:34.939498 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:34.939169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37192)
0620 02:24:34.939325 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:34.939393 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 02:24:34.939708 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
I20260620 02:24:35.045920 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 3
I20260620 02:24:35.047027 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.046553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37196)
0620 02:24:35.046753 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.046861 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:24:35.048229 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.047845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37200)
0620 02:24:35.048040 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.048106 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 02:24:35.048424 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
I20260620 02:24:35.052143 23816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 4
I20260620 02:24:35.053222 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.052762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37210)
0620 02:24:35.052971 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.053065 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 02:24:35.053669 23824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:35.054005 23824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:35.054160 23824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 02:24:35.054215 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.053848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37218)
0620 02:24:35.053997 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.054071 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:24:35.054420 23816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:36755: Network error: Client connection negotiation failed: client connection to 127.9.231.124:36755: connect: Connection refused (error 111)
W20260620 02:24:35.061820 23824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:35.061995 23824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:35.062095 23824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:35.062199 23824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:35.062285 23824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:35.072862 23824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:40695,127.9.231.125:42775,127.9.231.124:36755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab
--principal=kudu/127.9.231.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:36755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.231.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:35.074339 23824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:35.075958 23824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:35.089380 23830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:35.089401 23829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:35.089792 23832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:35.090368 23824 server_base.cc:1034] running on GCE node
I20260620 02:24:35.091266 23824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:35.092866 23824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:35.094087 23824 hybrid_clock.cc:648] HybridClock initialized: now 1781922275094029 us; error 51 us; skew 500 ppm
I20260620 02:24:35.094486 23824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:35.097811 23824 init.cc:377] Logged in from keytab as kudu/127.9.231.124@KRBTEST.COM (short username kudu)
I20260620 02:24:35.099504 23824 webserver.cc:460] Webserver started at http://127.9.231.124:43523/ using document root <none> and password file <none>
I20260620 02:24:35.100291 23824 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:35.100447 23824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:35.100787 23824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:35.104033 23824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data/instance:
uuid: "592282fc8c744162be7c3f696ac8f767"
format_stamp: "Formatted at 2026-06-20 02:24:35 on dist-test-slave-9vg1"
server_key: "299be31e89198acfadade5ff94e91906"
server_key_iv: "ba1a59c06429ecf7745ce2a4eb8fa53d"
server_key_version: "encryptionkey@0"
I20260620 02:24:35.104938 23824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/wal/instance:
uuid: "592282fc8c744162be7c3f696ac8f767"
format_stamp: "Formatted at 2026-06-20 02:24:35 on dist-test-slave-9vg1"
server_key: "299be31e89198acfadade5ff94e91906"
server_key_iv: "ba1a59c06429ecf7745ce2a4eb8fa53d"
server_key_version: "encryptionkey@0"
I20260620 02:24:35.110630 23824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:24:35.114530 23839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:35.116012 23824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:24:35.116216 23824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/wal
uuid: "592282fc8c744162be7c3f696ac8f767"
format_stamp: "Formatted at 2026-06-20 02:24:35 on dist-test-slave-9vg1"
server_key: "299be31e89198acfadade5ff94e91906"
server_key_iv: "ba1a59c06429ecf7745ce2a4eb8fa53d"
server_key_version: "encryptionkey@0"
I20260620 02:24:35.116441 23824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:35.135800 23824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:35.140467 23824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:35.140789 23824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:35.161037 23824 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.124:36755
I20260620 02:24:35.161129 23890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.124:36755 every 8 connection(s)
I20260620 02:24:35.162500 23824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
I20260620 02:24:35.166482 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:35.171754 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23824
I20260620 02:24:35.171979 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922055280103-10141-0/minicluster-data/master-2/wal/instance
I20260620 02:24:35.172495 10141 external_mini_cluster.cc:1428] Setting key 03b1c934a333a0e58787cfd5bec3332c
I20260620 02:24:35.173739 23891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:24:35.196468 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.176761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.9.231.126:40695)
0620 02:24:35.177269 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.177277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.177965 (+ 688us) server_negotiation.cc:407] Connection header received
0620 02:24:35.178448 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.178457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.178544 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.178650 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.180568 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.181833 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.184215 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.184872 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.185003 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.189398 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.189407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.189414 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.189477 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.192751 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.193604 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.193613 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.193617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.193857 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.194300 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.194305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.194311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.194715 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.194831 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.195807 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.196042 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
I20260620 02:24:35.197651 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.176810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40695 (local address 127.0.0.1:54214)
0620 02:24:35.177790 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.177845 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:35.178185 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:35.178780 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:35.178787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:35.179719 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:35.180333 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.180364 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.182016 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.182020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.183979 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.183999 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.185104 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.185108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.185331 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.186214 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:35.186257 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:35.189153 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:35.192899 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.192917 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.192932 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.193459 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.193977 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.193981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.193985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.194150 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.194839 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:35.194847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:35.195329 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 02:24:35.195628 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:24:35.196154 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":111,"spinlock_wait_cycles":10624,"thread_start_us":538,"threads_started":1}
I20260620 02:24:35.197700 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.174570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.9.231.124:36755)
0620 02:24:35.175516 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.175528 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.175572 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:24:35.177885 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.177922 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.178315 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.178582 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.180572 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.182285 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.184214 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.185150 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.185323 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.190211 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.190221 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.190235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.190268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.193107 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.193887 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.193894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.193896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.194110 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.194570 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.194573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.194575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.195322 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.195620 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.195982 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.196246 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":717,"spinlock_wait_cycles":28288,"thread_start_us":466,"threads_started":1}
I20260620 02:24:35.199218 23891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:35.214553 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.199896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:42775 (local address 127.0.0.1:58584)
0620 02:24:35.200054 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.200072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:35.200206 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:35.200622 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:35.200627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:35.201035 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:35.201236 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.201253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.202721 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.202725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.204108 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.204124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.205116 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.205121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.205294 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.205982 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:35.206005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:35.208353 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:35.211905 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.211911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.211915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.212351 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.212767 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.212773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.212776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.212925 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.213444 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:35.213449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:35.213631 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 02:24:35.213804 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 02:24:35.214144 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 02:24:35.214586 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.200137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.9.231.125:42775)
0620 02:24:35.200326 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.200333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.200354 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:24:35.200414 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.200418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.200477 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.200555 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.201411 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.202566 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.204286 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.204963 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.205100 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.208571 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.208580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.208587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.208637 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.211753 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.212489 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.212500 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.212502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.212647 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.213038 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.213044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.213047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.213323 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.213490 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.213955 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.214184 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:24:35.215840 23891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:24:35.217378 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.200982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.9.231.126:40695)
0620 02:24:35.201150 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.201156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.201189 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:24:35.201336 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.201343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.201430 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.201512 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.203030 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.204131 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.205937 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.206570 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.206698 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.211348 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.211353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.211357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.211394 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.214146 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.215113 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.215118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.215121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.215278 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.215729 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.215733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.215736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.216106 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.216353 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.216733 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.217035 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:24:35.225651 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 4
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:24:35.233160 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.216999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.9.231.124:36755)
0620 02:24:35.217171 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.217177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.217297 (+ 120us) server_negotiation.cc:407] Connection header received
0620 02:24:35.217464 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.217470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.217561 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.217648 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.218464 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.219450 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.220854 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.221564 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.221654 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.224872 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.224879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.224883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.224926 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.228843 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.229706 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.229712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.229716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.229883 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.231544 (+ 1661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.231549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.231553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.231889 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.232048 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.232426 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.232828 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":40}
I20260620 02:24:35.233160 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.216749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37244)
0620 02:24:35.217206 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.217225 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:35.217334 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:35.217722 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:35.217727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:35.218191 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:35.218325 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.218333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.219583 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.219589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.220712 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.220724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.221718 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.221724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.221918 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.222521 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:35.222547 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:35.224649 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:35.229032 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.229038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.229042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.229566 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.231202 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.231208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.231212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.231414 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.232017 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:35.232021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:35.232177 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 02:24:35.232343 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 02:24:35.232685 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":4}
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922273, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:24:35.237895 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.219304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.9.231.125:42775)
0620 02:24:35.219470 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.219474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.219554 (+ 80us) server_negotiation.cc:407] Connection header received
0620 02:24:35.219708 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.219714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.219787 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.219888 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.221402 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.222426 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.223770 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.224314 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.224457 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.229970 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.229979 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.229983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.230031 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.234284 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.235502 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.235508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.235511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.235706 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.236188 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.236194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.236197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.236580 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.236924 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.237257 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.237509 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:24:35.240267 23891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767: Bootstrap starting.
I20260620 02:24:35.243804 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:24:35.244071 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.226357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37256)
0620 02:24:35.226826 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.226847 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:35.226965 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:35.227757 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:35.227764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:35.228205 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:35.228361 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.228369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.230038 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.230044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.231030 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.231042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.232477 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.232484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.232723 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.233332 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:35.233364 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:35.236178 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:35.240975 (+ 4797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.240982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.240988 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.241465 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.241902 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.241908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.241911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.242100 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.242891 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:35.242896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:35.243141 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 02:24:35.243322 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 02:24:35.243675 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313}
I20260620 02:24:35.244707 23891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:35.245270 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.226430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.9.231.124:36755)
0620 02:24:35.227463 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.227467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.227486 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:24:35.227529 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.227532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.227589 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.227732 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.228527 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.229891 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.231237 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.232285 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.234853 (+ 2568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.236427 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.236438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.236443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.236476 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.240763 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.241617 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.241621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.241624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.241775 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.242260 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.242265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.242269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.242718 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.242903 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.244514 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.244794 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":685,"threads_started":1}
I20260620 02:24:35.246246 23891 log.cc:824] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:35.250377 23891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767: No bootstrap required, opened a new log
I20260620 02:24:35.250880 23742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: Bootstrap starting.
I20260620 02:24:35.255322 23742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:35.256740 23742 log.cc:824] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:35.260614 23742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: No bootstrap required, opened a new log
I20260620 02:24:35.261781 23891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.262267 23891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:35.262404 23891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592282fc8c744162be7c3f696ac8f767, State: Initialized, Role: FOLLOWER
I20260620 02:24:35.263199 23891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.272728 23742 raft_consensus.cc:348] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.273188 23742 raft_consensus.cc:374] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:35.273334 23742 raft_consensus.cc:729] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25692c655ab47338880f1dc3754a23b, State: Initialized, Role: FOLLOWER
I20260620 02:24:35.274353 23907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.274863 23907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:24:35.275523 23742 consensus_queue.cc:260] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.276446 23891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:35.285239 23742 sys_catalog.cc:564] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:35.282755 23908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.286641 23908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:24:35.287786 23907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:24:35.288136 23907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.288511 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.260431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.9.231.124:36755)
0620 02:24:35.260600 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.260606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.265008 (+ 4402us) server_negotiation.cc:407] Connection header received
0620 02:24:35.265286 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.265294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.265384 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.265452 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.268789 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.269840 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.272269 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.272912 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.273035 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.277376 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.277387 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.277395 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.277460 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.282358 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.284890 (+ 2532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.284896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.284899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.285076 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.285558 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.285563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.285566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.285999 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.286259 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.287856 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.288168 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:24:35.289866 23907 leader_election.cc:288] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b25692c655ab47338880f1dc3754a23b (127.9.231.126:40695), de6518f92b694cbb818fec2bfb9a7bac (127.9.231.125:42775)
I20260620 02:24:35.292071 23717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "592282fc8c744162be7c3f696ac8f767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25692c655ab47338880f1dc3754a23b" is_pre_election: true
I20260620 02:24:35.292909 23717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592282fc8c744162be7c3f696ac8f767 in term 0.
I20260620 02:24:35.293817 23843 leader_election.cc:302] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592282fc8c744162be7c3f696ac8f767, b25692c655ab47338880f1dc3754a23b; no voters:
I20260620 02:24:35.294297 23907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:24:35.294440 23907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:24:35.294541 23907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:35.297956 23907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.299715 23717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "592282fc8c744162be7c3f696ac8f767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25692c655ab47338880f1dc3754a23b"
I20260620 02:24:35.299829 23907 leader_election.cc:288] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [CANDIDATE]: Term 1 election: Requested vote from peers b25692c655ab47338880f1dc3754a23b (127.9.231.126:40695), de6518f92b694cbb818fec2bfb9a7bac (127.9.231.125:42775)
I20260620 02:24:35.300093 23717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:35.303571 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.258792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.9.231.126:40695)
0620 02:24:35.258987 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.258994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.259561 (+ 567us) server_negotiation.cc:407] Connection header received
0620 02:24:35.259822 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.259829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.259924 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.260043 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.267441 (+ 7398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.268956 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.274636 (+ 5680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.275526 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.275668 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.288190 (+ 12522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.288205 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.288214 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.288312 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.295973 (+ 7661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.297543 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.297549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.297577 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.297861 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.298847 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.298879 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.298884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.299830 (+ 946us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.300483 (+ 653us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.301829 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.302405 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 02:24:35.308665 23696 tablet.cc:2369] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:24:35.311586 23717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592282fc8c744162be7c3f696ac8f767 in term 1.
I20260620 02:24:35.312304 23843 leader_election.cc:302] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592282fc8c744162be7c3f696ac8f767, b25692c655ab47338880f1dc3754a23b; no voters:
I20260620 02:24:35.312655 23907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:24:35.318769 23907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [term 1 LEADER]: Becoming Leader. State: Replica: 592282fc8c744162be7c3f696ac8f767, State: Running, Role: LEADER
I20260620 02:24:35.319512 23907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.320165 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.261073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.9.231.125:42775)
0620 02:24:35.261343 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.261355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.264303 (+ 2948us) server_negotiation.cc:407] Connection header received
0620 02:24:35.264539 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.264550 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.264664 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.264799 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.267800 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.269055 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.277182 (+ 8127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.278049 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.278262 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.297794 (+ 19532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.297810 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.297821 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.297906 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.310426 (+ 12520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.313762 (+ 3336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.313773 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.313811 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.314259 (+ 448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.315215 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.315253 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.315259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.316243 (+ 984us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.318614 (+ 2371us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.318944 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.319452 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 02:24:35.330341 23921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 592282fc8c744162be7c3f696ac8f767. Latest consensus state: current_term: 1 leader_uuid: "592282fc8c744162be7c3f696ac8f767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.330796 23921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: This master's current role is: LEADER
W20260620 02:24:35.337278 23929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:24:35.337427 23929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:24:35.337545 23933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:24:35.339987 23933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:24:35.354221 23816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } attempt: 5
I20260620 02:24:35.335501 23824 master_runner.cc:186] Error getting master registration for 127.9.231.126:40695: OK, instance_id { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" instance_seqno: 1781922274069596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:24:35.360975 23824 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 02:24:35 dist-test-slave-9vg1 krb5kdc[23662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922274, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:24:35.358675 23675 master_runner.cc:186] Error getting master registration for 127.9.231.125:42775: OK, instance_id { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" instance_seqno: 1781922274622399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:24:35.377494 23675 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:24:35.393119 23845 tablet.cc:2369] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:24:35.399231 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.354907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:36755 (local address 127.0.0.1:37282)
0620 02:24:35.355169 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:35.355192 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:35.355311 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:35.356826 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:35.356831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:35.357432 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:35.357664 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.357678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.361328 (+ 3650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.361340 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.363781 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:35.363812 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.369869 (+ 6057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.369882 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:35.370337 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.371511 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:35.371558 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:35.376401 (+ 4843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:35.388764 (+ 12363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.388774 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.388782 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.395558 (+ 6776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.396256 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:35.396264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:35.396271 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:35.396511 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:35.397597 (+ 1086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:35.397605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:35.397860 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 02:24:35.398126 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 02:24:35.398583 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 02:24:35.400997 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:35.355632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.9.231.124:36755)
0620 02:24:35.356473 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:35.356481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:35.356504 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:24:35.356572 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:35.356579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:35.356664 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:35.356788 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:35.358382 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.359902 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.367216 (+ 7314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:35.368482 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:35.368703 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:35.382950 (+ 14247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:35.382962 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:35.382970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:35.383037 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:35.388424 (+ 5387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.395813 (+ 7389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.395823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.395830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.396074 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:35.396685 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:35.396693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:35.396699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:35.397235 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:35.397495 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:35.400033 (+ 2538us) server_negotiation.cc:299] Negotiation successful
0620 02:24:35.400412 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674}
W20260620 02:24:35.410782 23932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 02:24:35.419760 23816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac: Bootstrap starting.
W20260620 02:24:35.427310 23932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:24:35.431327 23816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:35.433911 23816 log.cc:824] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:35.439046 23717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [term 1 FOLLOWER]: Refusing update from remote peer 592282fc8c744162be7c3f696ac8f767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:35.445039 23921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:35.457973 23816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac: No bootstrap required, opened a new log
I20260620 02:24:35.470038 23816 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.470532 23816 raft_consensus.cc:374] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:35.470649 23816 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6518f92b694cbb818fec2bfb9a7bac, State: Initialized, Role: FOLLOWER
I20260620 02:24:35.471378 23816 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } }
I20260620 02:24:35.473865 23940 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.474252 23940 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:24:35.479406 23908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 592282fc8c744162be7c3f696ac8f767. Latest consensus state: current_term: 1 leader_uuid: "592282fc8c744162be7c3f696ac8f767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.479719 23908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:24:35.481060 23816 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:35.481971 23789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:35.482033 23935 mvcc.cc:205] Tried to move back new op lower bound from 7298753640181297152 to 7298753639714988032. Current Snapshot: MvccSnapshot[applied={T|T < 7298753640181297152}]
I20260620 02:24:35.486697 23789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [term 1 FOLLOWER]: Refusing update from remote peer 592282fc8c744162be7c3f696ac8f767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:35.487353 23791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "592282fc8c744162be7c3f696ac8f767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de6518f92b694cbb818fec2bfb9a7bac" is_pre_election: true
I20260620 02:24:35.487987 23791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592282fc8c744162be7c3f696ac8f767 in term 1.
I20260620 02:24:35.488941 23790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "592282fc8c744162be7c3f696ac8f767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de6518f92b694cbb818fec2bfb9a7bac"
I20260620 02:24:35.493070 23790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592282fc8c744162be7c3f696ac8f767 in term 1.
I20260620 02:24:35.494645 23921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:35.500275 23921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "592282fc8c744162be7c3f696ac8f767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.500540 23921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:35.500775 23938 mvcc.cc:205] Tried to move back new op lower bound from 7298753640181297152 to 7298753639714988032. Current Snapshot: MvccSnapshot[applied={T|T < 7298753640181297152}]
I20260620 02:24:35.501704 23933 catalog_manager.cc:1347] Generated new cluster ID: 09eee32359bb40b1ab08126a2f249cb0
I20260620 02:24:35.501796 23933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:24:35.518083 23908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "592282fc8c744162be7c3f696ac8f767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.518384 23908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:24:35.547361 23933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:24:35.550256 23933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:24:35.551620 23921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "592282fc8c744162be7c3f696ac8f767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.551891 23921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:35.551807 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "592282fc8c744162be7c3f696ac8f767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.552186 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:24:35.552517 23940 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 592282fc8c744162be7c3f696ac8f767. Latest consensus state: current_term: 1 leader_uuid: "592282fc8c744162be7c3f696ac8f767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25692c655ab47338880f1dc3754a23b" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40695 } } peers { permanent_uuid: "de6518f92b694cbb818fec2bfb9a7bac" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 42775 } } peers { permanent_uuid: "592282fc8c744162be7c3f696ac8f767" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 36755 } } }
I20260620 02:24:35.554200 23940 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:24:35.580951 23933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 592282fc8c744162be7c3f696ac8f767: Generated new TSK 0
I20260620 02:24:35.582684 23933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:24:35.606557 23958 catalog_manager.cc:1259] Loaded cluster ID: 09eee32359bb40b1ab08126a2f249cb0
I20260620 02:24:35.606675 23958 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac: loading cluster ID for follower catalog manager: success
I20260620 02:24:35.610432 23958 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac: acquiring CA information for follower catalog manager: success
I20260620 02:24:35.613206 23958 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de6518f92b694cbb818fec2bfb9a7bac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:24:36.433609 23932 catalog_manager.cc:1259] Loaded cluster ID: 09eee32359bb40b1ab08126a2f249cb0
I20260620 02:24:36.433770 23932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: loading cluster ID for follower catalog manager: success
I20260620 02:24:36.436537 23932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: acquiring CA information for follower catalog manager: success
I20260620 02:24:36.438338 23932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b25692c655ab47338880f1dc3754a23b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:24:36.746809 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23675
I20260620 02:24:36.748209 23737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:36.860567 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23675
I20260620 02:24:36.885270 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23749
I20260620 02:24:36.886379 23811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:37.001586 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23749
I20260620 02:24:37.023283 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23824
I20260620 02:24:37.024385 23886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:37.134785 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23824
2026-06-20T02:24:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5030 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:24:37 dist-test-slave-9vg1 krb5kdc[23969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:24:37 dist-test-slave-9vg1 krb5kdc[23969](info): set up 2 sockets
Jun 20 02:24:37 dist-test-slave-9vg1 krb5kdc[23969](info): commencing operation
krb5kdc: starting...
W20260620 02:24:38.260766 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:24:38 dist-test-slave-9vg1 krb5kdc[23969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:24:38.519268 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44311
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:38.870641 23982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:38.870963 23982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:38.871111 23982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:38.878736 23982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:38.878870 23982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:38.878916 23982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:38.878966 23982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:38.879012 23982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:38.889711 23982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:44311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:38.891245 23982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:38.892596 23982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:38.906971 23987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:38.907038 23990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:38.907015 23988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:38.907934 23982 server_base.cc:1034] running on GCE node
I20260620 02:24:38.908819 23982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:38.910406 23982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:38.911684 23982 hybrid_clock.cc:648] HybridClock initialized: now 1781922278911621 us; error 74 us; skew 500 ppm
I20260620 02:24:38.912139 23982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:38 dist-test-slave-9vg1 krb5kdc[23969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922278, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:38.915769 23982 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:24:38.917536 23982 webserver.cc:460] Webserver started at http://127.9.231.126:38173/ using document root <none> and password file <none>
I20260620 02:24:38.918322 23982 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:38.918481 23982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:38.918818 23982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:38.922179 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "4354e00948d446a0882174707a2887eb"
format_stamp: "Formatted at 2026-06-20 02:24:38 on dist-test-slave-9vg1"
server_key: "0fc7e9e6ebcdd78742cf82cbb1cc9311"
server_key_iv: "b2b934ed5e4a419b59ef5d82f73361df"
server_key_version: "encryptionkey@0"
I20260620 02:24:38.923041 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "4354e00948d446a0882174707a2887eb"
format_stamp: "Formatted at 2026-06-20 02:24:38 on dist-test-slave-9vg1"
server_key: "0fc7e9e6ebcdd78742cf82cbb1cc9311"
server_key_iv: "b2b934ed5e4a419b59ef5d82f73361df"
server_key_version: "encryptionkey@0"
I20260620 02:24:38.929051 23982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:24:38.933264 23997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:38.935004 23982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 02:24:38.935341 23982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "4354e00948d446a0882174707a2887eb"
format_stamp: "Formatted at 2026-06-20 02:24:38 on dist-test-slave-9vg1"
server_key: "0fc7e9e6ebcdd78742cf82cbb1cc9311"
server_key_iv: "b2b934ed5e4a419b59ef5d82f73361df"
server_key_version: "encryptionkey@0"
I20260620 02:24:38.935595 23982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:38.955018 23982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:38.960036 23982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:38.960350 23982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:38.979648 23982 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:44311
I20260620 02:24:38.979655 24048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:44311 every 8 connection(s)
I20260620 02:24:38.981124 23982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:24:38.984449 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 23982
I20260620 02:24:38.984704 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:24:38.985636 10141 external_mini_cluster.cc:1428] Setting key 25edc3ccc1e7fdad68e5a8e19be6b93b
I20260620 02:24:38.985924 24049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:39.002063 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb: Bootstrap starting.
Jun 20 02:24:38 dist-test-slave-9vg1 krb5kdc[23969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:39.006554 24049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:39.008047 24049 log.cc:824] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:39.012240 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb: No bootstrap required, opened a new log
I20260620 02:24:39.014088 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:38.987802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.9.231.126:44311)
0620 02:24:38.988776 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:38.988789 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:38.988835 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:24:38.990963 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:38.990989 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:38.991598 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:38.991978 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:38.993987 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:38.995457 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:38.997391 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:38.998223 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:38.998381 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:39.003853 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:39.003878 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:39.003902 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:39.003975 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:39.008299 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:39.009598 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:39.009607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:39.009615 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:39.009920 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:39.010421 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:39.010426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:39.010429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:39.011260 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:39.011583 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:39.012045 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 02:24:39.012538 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":468,"threads_started":1}
I20260620 02:24:39.019470 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:24:39.021716 24049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4354e00948d446a0882174707a2887eb" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44311 } }
I20260620 02:24:39.022166 24049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:39.022298 24049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4354e00948d446a0882174707a2887eb, State: Initialized, Role: FOLLOWER
I20260620 02:24:39.023051 24049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4354e00948d446a0882174707a2887eb" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44311 } }
I20260620 02:24:39.023428 24049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:24:39.023577 24049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:24:39.023785 24049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:39.026048 24049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4354e00948d446a0882174707a2887eb" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44311 } }
I20260620 02:24:39.026594 24049 leader_election.cc:302] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4354e00948d446a0882174707a2887eb; no voters:
I20260620 02:24:39.027300 24049 leader_election.cc:288] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:24:39.027395 24054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:24:39.028224 24054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [term 1 LEADER]: Becoming Leader. State: Replica: 4354e00948d446a0882174707a2887eb, State: Running, Role: LEADER
I20260620 02:24:39.028823 24054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4354e00948d446a0882174707a2887eb" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44311 } }
I20260620 02:24:39.029558 24049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:39.031936 24055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4354e00948d446a0882174707a2887eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4354e00948d446a0882174707a2887eb" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44311 } } }
I20260620 02:24:39.032462 24055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:39.032697 24056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4354e00948d446a0882174707a2887eb. Latest consensus state: current_term: 1 leader_uuid: "4354e00948d446a0882174707a2887eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4354e00948d446a0882174707a2887eb" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44311 } } }
I20260620 02:24:39.033020 24056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:39.035362 24060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:24:39.041391 24060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:24:39.051527 24060 catalog_manager.cc:1347] Generated new cluster ID: 6d90ec4f1cd74d9cbf5b2302d24a9fa9
I20260620 02:24:39.051673 24060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:24:39.083321 24060 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:24:39.084755 24060 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:24:39.096408 24060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4354e00948d446a0882174707a2887eb: Generated new TSK 0
I20260620 02:24:39.097400 24060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:24:39.133710 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 23982
I20260620 02:24:39.135218 24044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:39.235723 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 23982
2026-06-20T02:24:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2097 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:24:39 dist-test-slave-9vg1 krb5kdc[24077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:24:39 dist-test-slave-9vg1 krb5kdc[24077](info): set up 2 sockets
Jun 20 02:24:39 dist-test-slave-9vg1 krb5kdc[24077](info): commencing operation
krb5kdc: starting...
W20260620 02:24:40.396988 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:24:40 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:24:40.659404 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38605
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:41.017980 24090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:41.018312 24090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:41.018404 24090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:41.026108 24090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:41.026242 24090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:41.026291 24090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:41.026348 24090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:41.026394 24090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:41.037286 24090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:41.038755 24090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:41.040211 24090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:41.053098 24096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:41.053190 24095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:41.053212 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:41.053947 24090 server_base.cc:1034] running on GCE node
I20260620 02:24:41.054792 24090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:41.056357 24090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:41.057585 24090 hybrid_clock.cc:648] HybridClock initialized: now 1781922281057508 us; error 67 us; skew 500 ppm
I20260620 02:24:41.057977 24090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:41 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:41.061685 24090 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:24:41.063651 24090 webserver.cc:460] Webserver started at http://127.9.231.126:37619/ using document root <none> and password file <none>
I20260620 02:24:41.064508 24090 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:41.064626 24090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:41.064904 24090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:41.068342 24090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "bb883c61b2f048c9994c27b136d8bb74"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "dca97dac161e7170081f230aaef2506e"
server_key_iv: "f58b122afda701b6c4344978625630c8"
server_key_version: "encryptionkey@0"
I20260620 02:24:41.069192 24090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "bb883c61b2f048c9994c27b136d8bb74"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "dca97dac161e7170081f230aaef2506e"
server_key_iv: "f58b122afda701b6c4344978625630c8"
server_key_version: "encryptionkey@0"
I20260620 02:24:41.075217 24090 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 02:24:41.079389 24105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:41.081012 24090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:24:41.081239 24090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "bb883c61b2f048c9994c27b136d8bb74"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "dca97dac161e7170081f230aaef2506e"
server_key_iv: "f58b122afda701b6c4344978625630c8"
server_key_version: "encryptionkey@0"
I20260620 02:24:41.081475 24090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:41.106021 24090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:41.111152 24090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:41.111476 24090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:41.132229 24090 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38605
I20260620 02:24:41.132248 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38605 every 8 connection(s)
I20260620 02:24:41.133879 24090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:24:41.135972 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24090
I20260620 02:24:41.136245 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:24:41.136904 10141 external_mini_cluster.cc:1428] Setting key f68357863c345b5a2235092084d87a44
I20260620 02:24:41.138386 24157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:24:41 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:41.151821 24157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Bootstrap starting.
I20260620 02:24:41.155540 24157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:41.156751 24157 log.cc:824] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:41.160631 24157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: No bootstrap required, opened a new log
I20260620 02:24:41.161123 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:41.138999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.9.231.126:38605)
0620 02:24:41.139861 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:41.139873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:41.139916 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:24:41.141598 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:41.141623 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:41.142395 (+ 772us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:41.142881 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:41.144487 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:41.145886 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:41.147334 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:41.148011 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:41.148225 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:41.152491 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:41.152499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:41.152522 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:41.152566 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:41.155841 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:41.156660 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:41.156668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:41.156675 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:41.156863 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:41.157455 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:41.157465 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:41.157470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:41.158461 (+ 991us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:41.158820 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:41.159495 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 02:24:41.159989 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":377,"threads_started":1}
I20260620 02:24:41.169698 24157 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:41.170169 24157 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:41.170286 24157 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb883c61b2f048c9994c27b136d8bb74, State: Initialized, Role: FOLLOWER
I20260620 02:24:41.170866 24157 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:41.171159 24157 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:24:41.171315 24157 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:24:41.171494 24157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:41.174394 24157 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:41.174997 24157 leader_election.cc:302] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb883c61b2f048c9994c27b136d8bb74; no voters:
I20260620 02:24:41.175711 24157 leader_election.cc:288] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:24:41.175858 24162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:24:41.176692 24162 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 LEADER]: Becoming Leader. State: Replica: bb883c61b2f048c9994c27b136d8bb74, State: Running, Role: LEADER
I20260620 02:24:41.177280 24162 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:41.178138 24157 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:41.180366 24163 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb883c61b2f048c9994c27b136d8bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } } }
I20260620 02:24:41.181185 24163 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:41.180780 24164 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb883c61b2f048c9994c27b136d8bb74. Latest consensus state: current_term: 1 leader_uuid: "bb883c61b2f048c9994c27b136d8bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } } }
I20260620 02:24:41.181489 24164 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:41.182602 24169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:24:41.195719 24169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:24:41.205545 24169 catalog_manager.cc:1347] Generated new cluster ID: 88c110b2c63e4c18ab6a8d76f414d590
I20260620 02:24:41.205708 24169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:24:41.216089 24169 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:24:41.217496 24169 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:24:41.229768 24169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Generated new TSK 0
I20260620 02:24:41.230720 24169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:24:41.311172 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38605
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:41.674345 24183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:41.674664 24183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:41.674866 24183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:41.682756 24183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:41.682873 24183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:41.683123 24183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:24:41.693683 24183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:41.695041 24183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:41.696389 24183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:41.710721 24188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:41.710772 24191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:41.710714 24189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:41.716838 24183 server_base.cc:1034] running on GCE node
I20260620 02:24:41.717562 24183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:41.718676 24183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:41.719897 24183 hybrid_clock.cc:648] HybridClock initialized: now 1781922281719826 us; error 68 us; skew 500 ppm
I20260620 02:24:41.720243 24183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:41 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:41.723606 24183 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:24:41.725175 24183 webserver.cc:460] Webserver started at http://127.9.231.65:44517/ using document root <none> and password file <none>
I20260620 02:24:41.725890 24183 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:41.725991 24183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:41.726241 24183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:41.729279 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "c9993e61e8feb47a68433804af54d6d7"
server_key_iv: "b7f107d9d7ffcb6a035fdb2051e03a4a"
server_key_version: "encryptionkey@0"
I20260620 02:24:41.730136 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "c9993e61e8feb47a68433804af54d6d7"
server_key_iv: "b7f107d9d7ffcb6a035fdb2051e03a4a"
server_key_version: "encryptionkey@0"
I20260620 02:24:41.735692 24183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:24:41.739425 24198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:41.741111 24183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:24:41.741274 24183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "c9993e61e8feb47a68433804af54d6d7"
server_key_iv: "b7f107d9d7ffcb6a035fdb2051e03a4a"
server_key_version: "encryptionkey@0"
I20260620 02:24:41.741482 24183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:41.763602 24183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:41.768519 24183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:41.768784 24183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:41.769822 24183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:41.771446 24183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:41.771540 24183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:41.771632 24183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:41.771677 24183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:41.807601 24183 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:34959
I20260620 02:24:41.807651 24310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:34959 every 8 connection(s)
I20260620 02:24:41.809015 24183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:24:41.811713 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24183
I20260620 02:24:41.811954 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:24:41.812773 10141 external_mini_cluster.cc:1428] Setting key e3b3144bc2d49e504269122e857efcfd
Jun 20 02:24:41 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:41.834062 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:41.811997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:53571 (local address 127.9.231.126:38605)
0620 02:24:41.812533 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:41.812540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:41.813606 (+ 1066us) server_negotiation.cc:407] Connection header received
0620 02:24:41.815970 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:41.815975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:41.816046 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:41.816151 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:41.818511 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:41.819379 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:41.822009 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:41.822448 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:41.822556 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:41.826615 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:41.826624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:41.826630 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:41.826677 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:41.830314 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:41.831311 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:41.831320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:41.831326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:41.831493 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:41.831953 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:41.831960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:41.831966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:41.832371 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:41.832524 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:41.833305 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 02:24:41.833573 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":264,"threads_started":1}
I20260620 02:24:41.834753 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:41.812536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:53571)
0620 02:24:41.813418 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:41.813468 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:41.815629 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:41.816448 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:41.816464 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:41.817418 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:41.818286 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:41.818321 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:41.819561 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:41.819568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:41.821824 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:41.821844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:41.822705 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:41.822715 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:41.823053 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:41.823955 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:41.823981 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:41.826347 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:41.830507 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:41.830516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:41.830536 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:41.831168 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:41.831611 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:41.831614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:41.831616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:41.831819 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:41.832509 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:41.832516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:41.832880 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 02:24:41.833126 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:24:41.833553 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":331,"threads_started":1}
I20260620 02:24:41.836613 24311 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38605
I20260620 02:24:41.836954 24311 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:41.837774 24311 heartbeater.cc:511] Master 127.9.231.126:38605 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:24:41.840248 24122 ts_manager.cc:194] Registered new tserver with Master: f3965cb6a0e64b3f9f403e60c38a9dcf (127.9.231.65:34959)
I20260620 02:24:41.841905 24122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:53571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:24:41.876542 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38605
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:42.217108 24317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:42.217491 24317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:42.217665 24317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:42.225271 24317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:42.225450 24317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:42.225646 24317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:24:42.235785 24317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:42.237250 24317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:42.238672 24317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:42.252768 24322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:42.252764 24323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:42.252792 24325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:42.253592 24317 server_base.cc:1034] running on GCE node
I20260620 02:24:42.254256 24317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:42.255350 24317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:42.256578 24317 hybrid_clock.cc:648] HybridClock initialized: now 1781922282256512 us; error 62 us; skew 500 ppm
I20260620 02:24:42.256897 24317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:42 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922282, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:42.266678 24317 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:24:42.268525 24317 webserver.cc:460] Webserver started at http://127.9.231.66:46355/ using document root <none> and password file <none>
I20260620 02:24:42.269264 24317 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:42.269418 24317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:42.269734 24317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:42.273005 24317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "906e10f5d1eb4bb1b8acd194c2c0c885"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "2299bdea03ec680d93dd65343c2e845e"
server_key_iv: "e58a340241737e439ad2d2db8fcb3f7a"
server_key_version: "encryptionkey@0"
I20260620 02:24:42.273867 24317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "906e10f5d1eb4bb1b8acd194c2c0c885"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "2299bdea03ec680d93dd65343c2e845e"
server_key_iv: "e58a340241737e439ad2d2db8fcb3f7a"
server_key_version: "encryptionkey@0"
I20260620 02:24:42.279901 24317 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:24:42.284462 24332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:42.286260 24317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:24:42.286424 24317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "906e10f5d1eb4bb1b8acd194c2c0c885"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "2299bdea03ec680d93dd65343c2e845e"
server_key_iv: "e58a340241737e439ad2d2db8fcb3f7a"
server_key_version: "encryptionkey@0"
I20260620 02:24:42.286614 24317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:42.322306 24317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:42.326747 24317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:42.327004 24317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:42.328058 24317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:42.329456 24317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:42.329533 24317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:42.329627 24317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:42.329682 24317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:42.364367 24317 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:45817
I20260620 02:24:42.364416 24444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:45817 every 8 connection(s)
I20260620 02:24:42.365705 24317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:24:42.372853 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24317
I20260620 02:24:42.373062 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:24:42.373618 10141 external_mini_cluster.cc:1428] Setting key 08b397c029c64227b9f74f1e1604ae74
Jun 20 02:24:42 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922282, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:42.389264 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.368475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:34281 (local address 127.9.231.126:38605)
0620 02:24:42.368951 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:42.368955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:42.369817 (+ 862us) server_negotiation.cc:407] Connection header received
0620 02:24:42.371792 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:42.371798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:42.371897 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:42.372026 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:42.374205 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.375162 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.377516 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.377999 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.378099 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:42.382562 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:42.382567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:42.382571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:42.382608 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:42.385852 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:42.386699 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:42.386707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:42.386711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:42.386859 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:42.387388 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:42.387393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:42.387396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:42.387687 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:42.387845 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:42.388608 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 02:24:42.388863 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":260,"threads_started":1}
I20260620 02:24:42.390404 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.368939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:34281)
0620 02:24:42.369642 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:42.369691 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:42.371489 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:42.372299 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:42.372307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:42.373137 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:42.373998 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.374025 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.375330 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.375335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:42.377334 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.377355 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.378268 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.378278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:42.378615 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:42.379412 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:42.379454 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:42.382283 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:42.386036 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:42.386046 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:42.386067 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:42.386554 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:42.386982 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:42.386988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:42.386991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:42.387270 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:42.387832 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:42.387841 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:42.388235 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 02:24:42.388515 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 02:24:42.389119 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":283,"threads_started":1}
I20260620 02:24:42.392118 24445 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38605
I20260620 02:24:42.392479 24445 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:42.393483 24445 heartbeater.cc:511] Master 127.9.231.126:38605 requested a full tablet report, sending...
I20260620 02:24:42.395231 24122 ts_manager.cc:194] Registered new tserver with Master: 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817)
I20260620 02:24:42.396098 24122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:34281
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:24:42.434082 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38605
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:42.778168 24451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:42.778481 24451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:42.778673 24451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:42.786976 24451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:42.787174 24451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:42.787297 24451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:24:42.797783 24451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:42.799213 24451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:42.800576 24451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:42.816759 24459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:42.816792 24456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:42.817065 24457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:42.818742 24451 server_base.cc:1034] running on GCE node
I20260620 02:24:42.819609 24451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:42.820681 24451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:42.821905 24451 hybrid_clock.cc:648] HybridClock initialized: now 1781922282821832 us; error 60 us; skew 500 ppm
I20260620 02:24:42.822252 24451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:42 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922282, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:42.825678 24451 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:24:42.827361 24451 webserver.cc:460] Webserver started at http://127.9.231.67:34817/ using document root <none> and password file <none>
I20260620 02:24:42.828110 24451 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:42.828243 24451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:42.828583 24451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:24:42.831724 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "9f6a9989ef164361a6cac149f4b14824"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "3967071d4e1b00fef949ca9efe706bc2"
server_key_iv: "4061d196e24362ea3c65cba2ca925989"
server_key_version: "encryptionkey@0"
I20260620 02:24:42.832540 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "9f6a9989ef164361a6cac149f4b14824"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "3967071d4e1b00fef949ca9efe706bc2"
server_key_iv: "4061d196e24362ea3c65cba2ca925989"
server_key_version: "encryptionkey@0"
I20260620 02:24:42.838274 24451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:24:42.842226 24466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:42.844002 24451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:24:42.844240 24451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "9f6a9989ef164361a6cac149f4b14824"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "3967071d4e1b00fef949ca9efe706bc2"
server_key_iv: "4061d196e24362ea3c65cba2ca925989"
server_key_version: "encryptionkey@0"
I20260620 02:24:42.844498 24451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:42.846134 24311 heartbeater.cc:503] Master 127.9.231.126:38605 was elected leader, sending a full tablet report...
I20260620 02:24:42.862829 24451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:42.868348 24451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:42.868700 24451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:42.869886 24451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:42.871476 24451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:42.871560 24451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:42.871752 24451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:42.871824 24451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:42.907552 24451 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:41795
I20260620 02:24:42.907590 24578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:41795 every 8 connection(s)
I20260620 02:24:42.908986 24451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:24:42.910830 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24451
I20260620 02:24:42.911027 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:24:42.911669 10141 external_mini_cluster.cc:1428] Setting key 134d2d3764312ad4d363e0b4d45a41e8
Jun 20 02:24:42 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922282, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:42.932837 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.912199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:59711 (local address 127.9.231.126:38605)
0620 02:24:42.912815 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:42.912821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:42.913943 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 02:24:42.916016 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:42.916021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:42.916100 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:42.916217 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:42.918529 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.919836 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.921846 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.922277 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.922430 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:42.926567 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:42.926575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:42.926585 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:42.926624 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:42.929673 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:42.930512 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:42.930516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:42.930518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:42.930643 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:42.931143 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:42.931146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:42.931148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:42.931427 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:42.931548 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:42.932370 (+ 822us) server_negotiation.cc:299] Negotiation successful
0620 02:24:42.932557 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":337,"threads_started":1}
I20260620 02:24:42.934011 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.912877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:59711)
0620 02:24:42.913771 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:42.913818 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:42.915744 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:42.916555 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:42.916563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:42.917319 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:42.918275 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.918305 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.920062 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.920069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:42.921650 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.921668 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.922543 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.922551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:42.922833 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:42.923578 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:42.923604 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:42.926319 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:42.929839 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:42.929849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:42.929865 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:42.930394 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:42.930746 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:42.930750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:42.930753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:42.930973 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:42.931548 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:42.931555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:42.932009 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 02:24:42.932284 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 02:24:42.932761 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":365,"threads_started":1}
I20260620 02:24:42.935775 24579 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38605
I20260620 02:24:42.936106 24579 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:42.936903 24579 heartbeater.cc:511] Master 127.9.231.126:38605 requested a full tablet report, sending...
I20260620 02:24:42.938782 24122 ts_manager.cc:194] Registered new tserver with Master: 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795)
I20260620 02:24:42.939936 24122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:59711
I20260620 02:24:42.941222 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:24:42.959709 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.946463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.9.231.126:38605)
0620 02:24:42.946845 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:42.946850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:42.947183 (+ 333us) server_negotiation.cc:407] Connection header received
0620 02:24:42.947331 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:42.947335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:42.947402 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:42.947491 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:42.948899 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.949800 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.951298 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.951718 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.951817 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:42.954287 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:42.954293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:42.954296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:42.954330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:42.957021 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:42.957693 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:42.957697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:42.957699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:42.957835 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:42.958195 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:42.958199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:42.958201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:42.958638 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:42.958833 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:42.959216 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:24:42.959408 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:24:42.964285 24122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:24:42.971599 24122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:24:43.002043 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.986586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.9.231.67:41795)
0620 02:24:42.991616 (+ 5030us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:42.991630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:42.991683 (+ 53us) server_negotiation.cc:407] Connection header received
0620 02:24:42.991836 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:42.991849 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:42.992216 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:42.993452 (+ 1236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:42.993810 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.995959 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.998851 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.000620 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.000771 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.001020 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.001351 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.001623 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4766,"thread_start_us":4553,"threads_started":1}
I20260620 02:24:43.002779 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.986289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.9.231.66:45817)
0620 02:24:42.992694 (+ 6405us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:42.992707 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:42.992744 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:24:42.992865 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:42.992877 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:42.993182 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:42.993428 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:42.994580 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.996647 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.999655 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.001347 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.001492 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.001676 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.002309 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.002495 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6194,"thread_start_us":8934,"threads_started":1}
I20260620 02:24:43.002782 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.985747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:41795 (local address 127.0.0.1:36592)
0620 02:24:42.986967 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:42.987021 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:42.987308 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:42.992528 (+ 5220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:42.992537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:42.992582 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:42.993126 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.993146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.996199 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.996208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:42.998646 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.998664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.000844 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.000851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.001065 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.001113 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 02:24:43.001259 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:24:43.001997 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":85,"thread_start_us":5946,"threads_started":1}
I20260620 02:24:43.003728 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.986027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:45817 (local address 127.0.0.1:35646)
0620 02:24:42.991994 (+ 5967us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:42.992017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:42.992155 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:42.993763 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:42.993771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:42.993808 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:42.994342 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.994357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:42.996819 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:42.996825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:42.999451 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.999467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.001834 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.001841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.002064 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.002111 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 02:24:43.003085 (+ 974us) client_negotiation.cc:240] Negotiation successful
0620 02:24:43.003255 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5812,"mutex_wait_us":67,"thread_start_us":916,"threads_started":1}
I20260620 02:24:43.006395 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.992426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:34959 (local address 127.0.0.1:34710)
0620 02:24:42.998419 (+ 5993us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:42.998439 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:42.998571 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:42.999488 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:42.999495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:42.999521 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:42.999850 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:42.999858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.001553 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.001558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.003934 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:43.003949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.005573 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.005579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.005751 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.005792 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:24:43.005889 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:24:43.006029 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5842,"thread_start_us":5654,"threads_started":1}
I20260620 02:24:43.007038 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:42.992553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.9.231.65:34959)
0620 02:24:42.996978 (+ 4425us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:42.996989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:42.998718 (+ 1729us) server_negotiation.cc:407] Connection header received
0620 02:24:42.998846 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:42.998854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:42.999188 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:42.999416 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:43.000079 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.001371 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.004116 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.005332 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.005513 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.005755 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.006598 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.006769 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4232,"thread_start_us":4052,"threads_started":1}
I20260620 02:24:43.007560 24514 tablet_service.cc:1467] Processing CreateTablet for tablet fd112f4b0d9941db943b6f4d600942b7 (DEFAULT_TABLE table=test-table [id=135dfb9129874aadba45dd43892cfb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:43.007560 24380 tablet_service.cc:1467] Processing CreateTablet for tablet fd112f4b0d9941db943b6f4d600942b7 (DEFAULT_TABLE table=test-table [id=135dfb9129874aadba45dd43892cfb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:43.009344 24514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd112f4b0d9941db943b6f4d600942b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:43.009949 24380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd112f4b0d9941db943b6f4d600942b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:43.011170 24246 tablet_service.cc:1467] Processing CreateTablet for tablet fd112f4b0d9941db943b6f4d600942b7 (DEFAULT_TABLE table=test-table [id=135dfb9129874aadba45dd43892cfb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:43.012445 24246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd112f4b0d9941db943b6f4d600942b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:43.025321 24600 tablet_bootstrap.cc:492] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Bootstrap starting.
I20260620 02:24:43.029676 24598 tablet_bootstrap.cc:492] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Bootstrap starting.
I20260620 02:24:43.030663 24599 tablet_bootstrap.cc:492] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Bootstrap starting.
I20260620 02:24:43.033149 24598 tablet_bootstrap.cc:654] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:43.033591 24599 tablet_bootstrap.cc:654] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:43.034665 24600 tablet_bootstrap.cc:654] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:43.035086 24598 log.cc:824] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:43.035156 24599 log.cc:824] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:43.036067 24600 log.cc:824] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:43.039746 24598 tablet_bootstrap.cc:492] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: No bootstrap required, opened a new log
I20260620 02:24:43.039885 24600 tablet_bootstrap.cc:492] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: No bootstrap required, opened a new log
I20260620 02:24:43.040091 24599 tablet_bootstrap.cc:492] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: No bootstrap required, opened a new log
I20260620 02:24:43.040105 24598 ts_tablet_manager.cc:1397] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 02:24:43.040192 24600 ts_tablet_manager.cc:1397] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260620 02:24:43.040369 24599 ts_tablet_manager.cc:1397] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260620 02:24:43.049469 24598 raft_consensus.cc:348] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.049937 24598 raft_consensus.cc:374] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:43.050040 24598 raft_consensus.cc:729] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3965cb6a0e64b3f9f403e60c38a9dcf, State: Initialized, Role: FOLLOWER
I20260620 02:24:43.051103 24598 consensus_queue.cc:260] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.051725 24599 raft_consensus.cc:348] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.051726 24600 raft_consensus.cc:348] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.052196 24599 raft_consensus.cc:374] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:43.052197 24600 raft_consensus.cc:374] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:43.052310 24599 raft_consensus.cc:729] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906e10f5d1eb4bb1b8acd194c2c0c885, State: Initialized, Role: FOLLOWER
I20260620 02:24:43.052306 24600 raft_consensus.cc:729] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6a9989ef164361a6cac149f4b14824, State: Initialized, Role: FOLLOWER
I20260620 02:24:43.053102 24598 ts_tablet_manager.cc:1428] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260620 02:24:43.053046 24599 consensus_queue.cc:260] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.053046 24600 consensus_queue.cc:260] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.054441 24579 heartbeater.cc:503] Master 127.9.231.126:38605 was elected leader, sending a full tablet report...
I20260620 02:24:43.054932 24445 heartbeater.cc:503] Master 127.9.231.126:38605 was elected leader, sending a full tablet report...
I20260620 02:24:43.055173 24600 ts_tablet_manager.cc:1428] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 02:24:43.056525 24599 ts_tablet_manager.cc:1428] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260620 02:24:43.061640 24312 tablet.cc:2369] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:24:43.117807 24446 tablet.cc:2369] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:24:43.135318 24604 raft_consensus.cc:482] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:24:43.135736 24604 raft_consensus.cc:504] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.137638 24604 leader_election.cc:288] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795), 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817)
I20260620 02:24:43.146543 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.138907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:56173 (local address 127.9.231.66:45817)
0620 02:24:43.139125 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:43.139130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:43.139333 (+ 203us) server_negotiation.cc:407] Connection header received
0620 02:24:43.139618 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:43.139627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:43.139716 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:43.139824 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:43.140616 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.141653 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.144037 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.145473 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.145673 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.145754 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.146099 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.146270 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:24:43.146723 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.138215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:47465 (local address 127.9.231.67:41795)
0620 02:24:43.138643 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:43.138650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:43.139029 (+ 379us) server_negotiation.cc:407] Connection header received
0620 02:24:43.139420 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:43.139426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:43.139528 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:43.139634 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:43.140351 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.141509 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.144286 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.145518 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.145673 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.145754 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.146318 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.146459 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260620 02:24:43.146708 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.138083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:45817 (local address 127.9.231.65:56173)
0620 02:24:43.139235 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:43.139257 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:43.139438 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:43.139875 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:43.139879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:43.139909 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:43.140410 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:43.140423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.141824 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.141831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.143769 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:43.143784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.145699 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.145706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.145884 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.145951 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 02:24:43.146061 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:24:43.146217 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":61,"thread_start_us":1037,"threads_started":1}
I20260620 02:24:43.147661 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.137817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:41795 (local address 127.9.231.65:47465)
0620 02:24:43.138892 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:43.138914 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:43.139197 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:43.139671 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:43.139676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:43.139701 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:43.140182 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:43.140199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.141667 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.141673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.144126 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:43.144144 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.145910 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.145940 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:43.146124 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.146157 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:24:43.147135 (+ 978us) client_negotiation.cc:240] Negotiation successful
0620 02:24:43.147282 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":108,"thread_start_us":723,"threads_started":1}
I20260620 02:24:43.148166 24400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd112f4b0d9941db943b6f4d600942b7" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" is_pre_election: true
I20260620 02:24:43.148842 24400 raft_consensus.cc:2454] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 0.
I20260620 02:24:43.148826 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd112f4b0d9941db943b6f4d600942b7" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6a9989ef164361a6cac149f4b14824" is_pre_election: true
I20260620 02:24:43.149411 24534 raft_consensus.cc:2454] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 0.
I20260620 02:24:43.149641 24202 leader_election.cc:302] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906e10f5d1eb4bb1b8acd194c2c0c885, f3965cb6a0e64b3f9f403e60c38a9dcf; no voters:
I20260620 02:24:43.150141 24604 raft_consensus.cc:2781] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:24:43.150295 24604 raft_consensus.cc:482] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:24:43.150406 24604 raft_consensus.cc:3037] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:43.152743 24604 raft_consensus.cc:504] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
I20260620 02:24:43.153388 24604 leader_election.cc:288] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 election: Requested vote from peers 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795), 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817)
I20260620 02:24:43.154016 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd112f4b0d9941db943b6f4d600942b7" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6a9989ef164361a6cac149f4b14824"
I20260620 02:24:43.154058 24400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd112f4b0d9941db943b6f4d600942b7" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885"
I20260620 02:24:43.154255 24534 raft_consensus.cc:3037] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:43.154290 24400 raft_consensus.cc:3037] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:43.156744 24400 raft_consensus.cc:2454] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 1.
I20260620 02:24:43.156745 24534 raft_consensus.cc:2454] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 1.
I20260620 02:24:43.157310 24202 leader_election.cc:302] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906e10f5d1eb4bb1b8acd194c2c0c885, f3965cb6a0e64b3f9f403e60c38a9dcf; no voters:
I20260620 02:24:43.157696 24604 raft_consensus.cc:2781] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:24:43.158221 24604 raft_consensus.cc:686] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 LEADER]: Becoming Leader. State: Replica: f3965cb6a0e64b3f9f403e60c38a9dcf, State: Running, Role: LEADER
I20260620 02:24:43.158812 24604 consensus_queue.cc:237] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } }
W20260620 02:24:43.160965 24580 tablet.cc:2369] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:24:43.164155 24121 catalog_manager.cc:5495] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf reported cstate change: term changed from 0 to 1, leader changed from <none> to f3965cb6a0e64b3f9f403e60c38a9dcf (127.9.231.65). New cstate: current_term: 1 leader_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } health_report { overall_health: HEALTHY } } }
I20260620 02:24:43.213129 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.207478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.9.231.65:34959)
0620 02:24:43.207750 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:43.207759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:43.207794 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:24:43.207904 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:43.207908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:43.207989 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:43.208149 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:24:43.208539 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.209448 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.210860 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.211331 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.211473 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.211985 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:24:43.212510 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:24:43.212690 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.212741 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.212826 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:24:43.222311 24400 raft_consensus.cc:1261] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Refusing update from remote peer f3965cb6a0e64b3f9f403e60c38a9dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:43.222311 24534 raft_consensus.cc:1261] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Refusing update from remote peer f3965cb6a0e64b3f9f403e60c38a9dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:43.223464 24609 consensus_queue.cc:1035] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:43.223879 24604 consensus_queue.cc:1035] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:43.237476 24614 mvcc.cc:205] Tried to move back new op lower bound from 7298753672066416640 to 7298753671823060992. Current Snapshot: MvccSnapshot[applied={T|T < 7298753672066416640}]
I20260620 02:24:43.242967 24612 mvcc.cc:205] Tried to move back new op lower bound from 7298753672066416640 to 7298753671823060992. Current Snapshot: MvccSnapshot[applied={T|T < 7298753672066416640}]
I20260620 02:24:43.257584 24120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:24:43.257968 24120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:24:43.262298 24120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Sending DeleteTablet for 3 replicas of tablet fd112f4b0d9941db943b6f4d600942b7
I20260620 02:24:43.264058 24514 tablet_service.cc:1514] Processing DeleteTablet for tablet fd112f4b0d9941db943b6f4d600942b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:43 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:36592
I20260620 02:24:43.264022 24380 tablet_service.cc:1514] Processing DeleteTablet for tablet fd112f4b0d9941db943b6f4d600942b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:43 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:35646
I20260620 02:24:43.264225 24246 tablet_service.cc:1514] Processing DeleteTablet for tablet fd112f4b0d9941db943b6f4d600942b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:43 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:34710
I20260620 02:24:43.265429 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24183
I20260620 02:24:43.274557 24625 tablet_replica.cc:330] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: stopping tablet replica
I20260620 02:24:43.275302 24625 raft_consensus.cc:2229] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:24:43.276294 24625 raft_consensus.cc:2258] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:43.277323 24623 tablet_replica.cc:330] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: stopping tablet replica
I20260620 02:24:43.277761 24624 tablet_replica.cc:330] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: stopping tablet replica
I20260620 02:24:43.278339 24624 raft_consensus.cc:2229] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:24:43.278861 24624 raft_consensus.cc:2258] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:43.279870 24623 raft_consensus.cc:2229] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:24:43.280400 24623 raft_consensus.cc:2258] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:43.280431 24625 ts_tablet_manager.cc:1905] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:43.281461 24624 ts_tablet_manager.cc:1905] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:43.282712 24623 ts_tablet_manager.cc:1905] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:43.289127 24625 ts_tablet_manager.cc:1918] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:43.289428 24625 log.cc:1196] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/fd112f4b0d9941db943b6f4d600942b7
I20260620 02:24:43.289570 24624 ts_tablet_manager.cc:1918] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:43.289815 24624 log.cc:1196] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/fd112f4b0d9941db943b6f4d600942b7
I20260620 02:24:43.290275 24623 ts_tablet_manager.cc:1918] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:43.290319 24625 ts_tablet_manager.cc:1939] T fd112f4b0d9941db943b6f4d600942b7 P f3965cb6a0e64b3f9f403e60c38a9dcf: Deleting consensus metadata
I20260620 02:24:43.290458 24623 log.cc:1196] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/fd112f4b0d9941db943b6f4d600942b7
I20260620 02:24:43.290470 24624 ts_tablet_manager.cc:1939] T fd112f4b0d9941db943b6f4d600942b7 P 906e10f5d1eb4bb1b8acd194c2c0c885: Deleting consensus metadata
I20260620 02:24:43.291082 24623 ts_tablet_manager.cc:1939] T fd112f4b0d9941db943b6f4d600942b7 P 9f6a9989ef164361a6cac149f4b14824: Deleting consensus metadata
I20260620 02:24:43.292646 24108 catalog_manager.cc:4841] TS f3965cb6a0e64b3f9f403e60c38a9dcf (127.9.231.65:34959): tablet fd112f4b0d9941db943b6f4d600942b7 (table test-table [id=135dfb9129874aadba45dd43892cfb14]) successfully deleted
I20260620 02:24:43.292634 24109 catalog_manager.cc:4841] TS 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817): tablet fd112f4b0d9941db943b6f4d600942b7 (table test-table [id=135dfb9129874aadba45dd43892cfb14]) successfully deleted
I20260620 02:24:43.293421 24108 catalog_manager.cc:4841] TS 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795): tablet fd112f4b0d9941db943b6f4d600942b7 (table test-table [id=135dfb9129874aadba45dd43892cfb14]) successfully deleted
Jun 20 02:24:43 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:24:43.305153 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.266967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.9.231.65:34959)
0620 02:24:43.267265 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:43.267276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:43.280078 (+ 12802us) server_negotiation.cc:407] Connection header received
0620 02:24:43.284279 (+ 4201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:43.284288 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:43.284410 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:43.284534 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:43.287376 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.288776 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.292477 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.293205 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.293375 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.298334 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:43.298353 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:43.298376 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:43.298447 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:43.301968 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:43.302794 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:43.302802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:43.302805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:43.302964 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:43.303423 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:43.303435 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:43.303437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:43.303857 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:43.304186 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.304521 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.304791 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 02:24:43.305902 24306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:43.427130 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24183
I20260620 02:24:43.447324 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24317
Jun 20 02:24:43 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:24:43.465080 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.448094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.9.231.66:45817)
0620 02:24:43.448310 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:43.448317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:43.448426 (+ 109us) server_negotiation.cc:407] Connection header received
0620 02:24:43.448663 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:43.448668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:43.448765 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:43.448891 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:43.450379 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.451382 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.452932 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.453390 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.453667 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.458508 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:43.458533 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:43.458550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:43.458593 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:43.461934 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:43.462799 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:43.462807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:43.462809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:43.462943 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:43.463407 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:43.463412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:43.463414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:43.463810 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:43.464131 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.464464 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.464707 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:24:43.465763 24440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:43.560096 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24317
I20260620 02:24:43.578198 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24451
Jun 20 02:24:43 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:24:43.594789 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:43.579201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.9.231.67:41795)
0620 02:24:43.579416 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:43.579421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:43.579442 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:24:43.579502 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:43.579506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:43.579572 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:43.579664 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:43.581025 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.581791 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.583181 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:43.583592 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:43.583700 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:43.588751 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:43.588770 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:43.588788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:43.588838 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:43.592001 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:43.592767 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:43.592775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:43.592777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:43.592923 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:43.593339 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:43.593343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:43.593346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:43.593761 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:43.594050 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:43.594322 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 02:24:43.594508 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:24:43.595711 24574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:43.689872 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24451
I20260620 02:24:43.707656 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24090
I20260620 02:24:43.708920 24152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:43.826062 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24090
I20260620 02:24:43.849339 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38605
--webserver_interface=127.9.231.126
--webserver_port=37619
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:44.217016 24631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:44.217355 24631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:44.217461 24631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:44.225391 24631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:44.225546 24631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:44.225600 24631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:44.225652 24631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:44.225706 24631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:44.238729 24631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=37619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:44.240386 24631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:44.241928 24631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:44.273453 24638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:44.273729 24637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:44.273592 24631 server_base.cc:1034] running on GCE node
W20260620 02:24:44.273443 24640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:44.275187 24631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:44.277035 24631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:44.278299 24631 hybrid_clock.cc:648] HybridClock initialized: now 1781922284278217 us; error 69 us; skew 500 ppm
I20260620 02:24:44.278777 24631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:44 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:44.282857 24631 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:24:44.285081 24631 webserver.cc:460] Webserver started at http://127.9.231.126:37619/ using document root <none> and password file <none>
I20260620 02:24:44.285970 24631 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:44.286159 24631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:44.292619 24631 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:24:44.296152 24647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:44.297991 24631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:24:44.298243 24631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "bb883c61b2f048c9994c27b136d8bb74"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "dca97dac161e7170081f230aaef2506e"
server_key_iv: "f58b122afda701b6c4344978625630c8"
server_key_version: "encryptionkey@0"
I20260620 02:24:44.299607 24631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:44.320955 24631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:44.325754 24631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:44.326066 24631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:44.346061 24631 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38605
I20260620 02:24:44.346081 24698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38605 every 8 connection(s)
I20260620 02:24:44.347766 24631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:24:44.354975 24699 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:24:44.355849 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24631
I20260620 02:24:44.356745 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:34959
--local_ip_for_outbound_sockets=127.9.231.65
--tserver_master_addrs=127.9.231.126:38605
--webserver_port=44517
--webserver_interface=127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:44.358916 24699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Bootstrap starting.
I20260620 02:24:44.387372 24699 log.cc:824] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:44.404894 24699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:24:44.405601 24699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Bootstrap complete.
I20260620 02:24:44.420094 24699 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:44.421418 24699 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb883c61b2f048c9994c27b136d8bb74, State: Initialized, Role: FOLLOWER
I20260620 02:24:44.422211 24699 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:44.422515 24699 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:24:44.422657 24699 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:24:44.422837 24699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:24:44.429813 24699 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:44.430558 24699 leader_election.cc:302] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb883c61b2f048c9994c27b136d8bb74; no voters:
I20260620 02:24:44.431356 24699 leader_election.cc:288] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:24:44.431622 24703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:24:44.432904 24703 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 LEADER]: Becoming Leader. State: Replica: bb883c61b2f048c9994c27b136d8bb74, State: Running, Role: LEADER
I20260620 02:24:44.433504 24703 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:44.433812 24699 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:44.436013 24705 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb883c61b2f048c9994c27b136d8bb74. Latest consensus state: current_term: 2 leader_uuid: "bb883c61b2f048c9994c27b136d8bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } } }
I20260620 02:24:44.436400 24705 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:44.445261 24704 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bb883c61b2f048c9994c27b136d8bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } } }
I20260620 02:24:44.445956 24708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:24:44.447156 24704 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:44.456174 24708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:24:44.457845 24708 catalog_manager.cc:1259] Loaded cluster ID: 88c110b2c63e4c18ab6a8d76f414d590
I20260620 02:24:44.457970 24708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:24:44.461445 24708 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 02:24:44.484814 24708 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 02:24:44.496042 24708 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:24:44.498178 24708 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Loaded TSK: 0
I20260620 02:24:44.498427 24721 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:24:44.500476 24708 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:24:44.765805 24701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:44.766153 24701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:44.766328 24701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:44.774019 24701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:44.774137 24701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:44.774308 24701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:24:44.785315 24701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:34959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=44517
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:44.786772 24701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:44.788270 24701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:44.804389 24727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:44.804662 24726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:44.805366 24729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:44.806290 24701 server_base.cc:1034] running on GCE node
I20260620 02:24:44.807044 24701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:44.808218 24701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:44.809453 24701 hybrid_clock.cc:648] HybridClock initialized: now 1781922284809391 us; error 58 us; skew 500 ppm
I20260620 02:24:44.809831 24701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:44 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:44.813180 24701 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:24:44.814848 24701 webserver.cc:460] Webserver started at http://127.9.231.65:44517/ using document root <none> and password file <none>
I20260620 02:24:44.815629 24701 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:44.815783 24701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:44.821918 24701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 02:24:44.825047 24736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:44.826653 24701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:24:44.826830 24701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "c9993e61e8feb47a68433804af54d6d7"
server_key_iv: "b7f107d9d7ffcb6a035fdb2051e03a4a"
server_key_version: "encryptionkey@0"
I20260620 02:24:44.827919 24701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:44.856691 24701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:44.866816 24701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:44.867177 24701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:44.869127 24701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:44.870836 24701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:44.870977 24701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:44.871106 24701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:44.871198 24701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:44.905519 24701 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:34959
I20260620 02:24:44.905578 24848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:34959 every 8 connection(s)
I20260620 02:24:44.907049 24701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:24:44.909631 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24701
I20260620 02:24:44.910460 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:45817
--local_ip_for_outbound_sockets=127.9.231.66
--tserver_master_addrs=127.9.231.126:38605
--webserver_port=46355
--webserver_interface=127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:24:44 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922284, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:44.940274 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:44.910556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:45905 (local address 127.9.231.126:38605)
0620 02:24:44.911678 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:44.911699 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:44.916145 (+ 4446us) server_negotiation.cc:407] Connection header received
0620 02:24:44.918687 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:44.918704 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:44.919241 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:44.919685 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:44.921420 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:44.922684 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:44.924403 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:44.925504 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:44.925708 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:44.931199 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:44.931211 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:44.931229 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:44.931280 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:44.935214 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:44.936341 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:44.936348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:44.936354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:44.936562 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:44.937106 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:44.937111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:44.937113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:44.937681 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:44.937838 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:44.938800 (+ 962us) server_negotiation.cc:299] Negotiation successful
0620 02:24:44.939347 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":702,"threads_started":1}
I20260620 02:24:44.940490 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:44.910826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:45905)
0620 02:24:44.915850 (+ 5024us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:44.915932 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:44.917981 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:44.919873 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:44.919884 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:44.920586 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:44.921156 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:44.921175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:44.922886 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:44.922890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:44.924242 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:44.924256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:44.925905 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:44.925918 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:44.926341 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:44.927318 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:44.927355 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:44.930716 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:44.935393 (+ 4677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:44.935404 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:44.935423 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:44.936165 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:44.936686 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:44.936691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:44.936695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:44.936941 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:44.937828 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:44.937837 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:44.938283 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 02:24:44.938579 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 02:24:44.939195 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4658,"thread_start_us":4343,"threads_started":1}
I20260620 02:24:44.943619 24849 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38605
I20260620 02:24:44.943987 24849 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:44.944823 24849 heartbeater.cc:511] Master 127.9.231.126:38605 requested a full tablet report, sending...
I20260620 02:24:44.947546 24664 ts_manager.cc:194] Registered new tserver with Master: f3965cb6a0e64b3f9f403e60c38a9dcf (127.9.231.65:34959)
I20260620 02:24:44.949565 24664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:45905
W20260620 02:24:45.267484 24853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:45.267822 24853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:45.267931 24853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:45.276093 24853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:45.276222 24853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:45.276340 24853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:24:45.286649 24853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:45817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=46355
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:45.288074 24853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:45.289412 24853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:45.304085 24861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:45.304503 24859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:45.304605 24858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:45.306222 24853 server_base.cc:1034] running on GCE node
I20260620 02:24:45.307106 24853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:45.308188 24853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:45.309418 24853 hybrid_clock.cc:648] HybridClock initialized: now 1781922285309370 us; error 34 us; skew 500 ppm
I20260620 02:24:45.309788 24853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:45 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922285, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:45.313491 24853 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:24:45.315320 24853 webserver.cc:460] Webserver started at http://127.9.231.66:46355/ using document root <none> and password file <none>
I20260620 02:24:45.316112 24853 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:45.316290 24853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:45.322213 24853 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 02:24:45.325407 24868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:45.326926 24853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:24:45.327176 24853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "906e10f5d1eb4bb1b8acd194c2c0c885"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "2299bdea03ec680d93dd65343c2e845e"
server_key_iv: "e58a340241737e439ad2d2db8fcb3f7a"
server_key_version: "encryptionkey@0"
I20260620 02:24:45.328275 24853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:45.348747 24853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:45.353818 24853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:45.354162 24853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:45.355336 24853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:45.357046 24853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:45.357200 24853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:45.357335 24853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:45.357420 24853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:45.394129 24853 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:45817
I20260620 02:24:45.394244 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:45817 every 8 connection(s)
I20260620 02:24:45.395823 24853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:24:45.398844 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24853
I20260620 02:24:45.399734 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:41795
--local_ip_for_outbound_sockets=127.9.231.67
--tserver_master_addrs=127.9.231.126:38605
--webserver_port=34817
--webserver_interface=127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:24:45 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922285, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:45.418239 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.399040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:47247 (local address 127.9.231.126:38605)
0620 02:24:45.399275 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:45.399280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:45.400820 (+ 1540us) server_negotiation.cc:407] Connection header received
0620 02:24:45.402772 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:45.402778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:45.402895 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:45.403048 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:45.404766 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.405486 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.407439 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.407962 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.408162 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.411983 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:45.411989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:45.411993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:45.412028 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:45.414788 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:45.415669 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:45.415674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:45.415684 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:45.415820 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:45.416306 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:45.416316 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:45.416318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:45.416607 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:45.416774 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:45.417595 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 02:24:45.417863 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:24:45.419405 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.399755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:47247)
0620 02:24:45.400631 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:45.400672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:45.402470 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:45.403378 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:45.403388 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:45.403989 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:45.404567 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.404585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.405683 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.405687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.407265 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.407283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.408282 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.408290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.408564 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.409339 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:45.409364 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:45.411719 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:45.414953 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:45.414963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:45.414982 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:45.415518 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:45.415948 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:45.415953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:45.415964 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:45.416175 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:45.416751 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:45.416761 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:45.417201 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 02:24:45.417480 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 02:24:45.418072 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":342,"threads_started":1}
I20260620 02:24:45.421157 24981 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38605
I20260620 02:24:45.421492 24981 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:45.422257 24981 heartbeater.cc:511] Master 127.9.231.126:38605 requested a full tablet report, sending...
I20260620 02:24:45.423961 24664 ts_manager.cc:194] Registered new tserver with Master: 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817)
I20260620 02:24:45.425212 24664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:47247
W20260620 02:24:45.751158 24984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:45.751502 24984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:45.751605 24984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:45.759407 24984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:45.759542 24984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:45.759657 24984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:24:45.770130 24984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:41795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=34817
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:45.771575 24984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:45.772974 24984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:45.788175 24989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:45.788189 24990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:45.788375 24992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:45.789206 24984 server_base.cc:1034] running on GCE node
I20260620 02:24:45.789884 24984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:45.791006 24984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:45.792299 24984 hybrid_clock.cc:648] HybridClock initialized: now 1781922285792247 us; error 37 us; skew 500 ppm
I20260620 02:24:45.792754 24984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:45 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922285, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:45.796597 24984 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:24:45.798239 24984 webserver.cc:460] Webserver started at http://127.9.231.67:34817/ using document root <none> and password file <none>
I20260620 02:24:45.799036 24984 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:45.799263 24984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:45.805395 24984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 02:24:45.808739 24999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:45.810334 24984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:24:45.810554 24984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "9f6a9989ef164361a6cac149f4b14824"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "3967071d4e1b00fef949ca9efe706bc2"
server_key_iv: "4061d196e24362ea3c65cba2ca925989"
server_key_version: "encryptionkey@0"
I20260620 02:24:45.811695 24984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:45.837028 24984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:45.841601 24984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:45.841889 24984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:45.843003 24984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:45.844606 24984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:45.844741 24984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:45.844878 24984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:45.844969 24984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:45.880981 24984 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:41795
I20260620 02:24:45.881075 25111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:41795 every 8 connection(s)
I20260620 02:24:45.882571 24984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:24:45.887389 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 24984
Jun 20 02:24:45 dist-test-slave-9vg1 krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922285, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:24:45.905459 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.888563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.9.231.126:38605)
0620 02:24:45.889343 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:45.889351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:45.889370 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:24:45.889431 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:45.889434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:45.889493 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:45.889646 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:45.891301 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.892147 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.893577 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.894050 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.894210 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.897405 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:45.897416 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:45.897420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:45.897455 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:45.901924 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:45.902811 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:45.902818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:45.902822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:45.903026 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:45.903591 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:45.903598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:45.903602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:45.904028 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:45.904433 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:45.904781 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 02:24:45.905026 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":648,"spinlock_wait_cycles":25984,"thread_start_us":504,"threads_started":1}
I20260620 02:24:45.909763 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.885335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:36209 (local address 127.9.231.126:38605)
0620 02:24:45.885552 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:45.885558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:45.886783 (+ 1225us) server_negotiation.cc:407] Connection header received
0620 02:24:45.889576 (+ 2793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:45.889582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:45.889662 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:45.889789 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:45.891459 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.892226 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.893809 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.894302 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.894442 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.898441 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:45.898448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:45.898453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:45.898570 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:45.902851 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:45.904075 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:45.904083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:45.904088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:45.904242 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:45.907422 (+ 3180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:45.907428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:45.907431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:45.907803 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:45.907997 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:45.909088 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0620 02:24:45.909371 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:24:45.910838 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.885844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:36209)
0620 02:24:45.886604 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:45.886650 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:45.889276 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:45.890056 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:45.890065 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:45.890600 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:24:45.891278 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.891300 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.892400 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.892404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.893643 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.893655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.894541 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.894550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.894789 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.895401 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:24:45.895425 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:24:45.898095 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:24:45.903082 (+ 4987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:45.903096 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:45.903119 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:45.903918 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:45.904416 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:24:45.904422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:24:45.904426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:24:45.907272 (+ 2846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:24:45.908123 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:24:45.908131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:24:45.908569 (+ 438us) client_negotiation.cc:769] Sending connection context
0620 02:24:45.908895 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 02:24:45.909491 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":298,"threads_started":1}
I20260620 02:24:45.912405 25112 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38605
I20260620 02:24:45.912739 25112 heartbeater.cc:464] Registering TS with master...
I20260620 02:24:45.913462 25112 heartbeater.cc:511] Master 127.9.231.126:38605 requested a full tablet report, sending...
I20260620 02:24:45.915042 24664 ts_manager.cc:194] Registered new tserver with Master: 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795)
I20260620 02:24:45.916115 24664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:36209
I20260620 02:24:45.923353 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:24:45.932451 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.924537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.9.231.126:38605)
0620 02:24:45.924728 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:45.924733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:45.925306 (+ 573us) server_negotiation.cc:407] Connection header received
0620 02:24:45.925605 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:45.925609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:45.925773 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:45.925924 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:24:45.926420 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.927385 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.929360 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.930107 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.930247 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.930909 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:24:45.931791 (+ 882us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:24:45.931989 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:45.932174 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 02:24:45.932251 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:24:45.933784 24664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:24:45.941525 24664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:24:45.953084 24849 heartbeater.cc:503] Master 127.9.231.126:38605 was elected leader, sending a full tablet report...
I20260620 02:24:45.968721 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.958685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.9.231.67:41795)
0620 02:24:45.961209 (+ 2524us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:45.961216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:45.961244 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:24:45.961318 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:45.961323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:45.961511 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:45.961726 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:45.962349 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.963947 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.965716 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.967251 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.967382 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.967669 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:45.968146 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 02:24:45.968339 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2346,"thread_start_us":2199,"threads_started":1}
I20260620 02:24:45.968847 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.959140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:41795 (local address 127.0.0.1:36610)
0620 02:24:45.960446 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:45.960469 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:45.960599 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:45.961710 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:45.961715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:45.961741 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:45.962124 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.962141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.964100 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.964105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.965550 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.965562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.967828 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.967834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.967977 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.968005 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:24:45.968094 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:24:45.968333 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"mutex_wait_us":4,"thread_start_us":1011,"threads_started":1}
I20260620 02:24:45.969803 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.955792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.9.231.65:34959)
0620 02:24:45.956487 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:45.956494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:45.958047 (+ 1553us) server_negotiation.cc:407] Connection header received
0620 02:24:45.958472 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:45.958481 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:45.958746 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:45.958922 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:45.960054 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.962569 (+ 2515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.965894 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.967389 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.968968 (+ 1579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.969232 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:45.969313 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 02:24:45.969487 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":350,"threads_started":1}
I20260620 02:24:45.968873 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.955585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:34959 (local address 127.0.0.1:34746)
0620 02:24:45.957873 (+ 2288us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:45.957922 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:45.958209 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:45.959296 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:45.959307 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:45.959349 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:45.959806 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.959822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.962726 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.962731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.965722 (+ 2991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.965737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.967645 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.967654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.967883 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.967946 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 02:24:45.968081 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 02:24:45.968333 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"mutex_wait_us":94,"thread_start_us":1846,"threads_started":1}
I20260620 02:24:45.972019 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.957471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:45817 (local address 127.0.0.1:35676)
0620 02:24:45.958180 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:45.958202 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:45.958316 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:45.963322 (+ 5006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:45.963327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:45.963354 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:45.963975 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.963988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.967170 (+ 3182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.967175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.969370 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:45.969383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.971216 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.971222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:45.971391 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.971433 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:24:45.971539 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:24:45.971679 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":1023,"threads_started":1}
I20260620 02:24:45.972436 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:45.957973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.9.231.66:45817)
0620 02:24:45.962442 (+ 4469us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:45.962453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:45.962482 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:24:45.962577 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:45.962586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:45.962840 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:45.963020 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:45.964206 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.965745 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.969532 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:45.970951 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:45.971129 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:45.971327 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:45.971593 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 02:24:45.971762 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4296,"thread_start_us":4174,"threads_started":1}
I20260620 02:24:45.973568 24784 tablet_service.cc:1467] Processing CreateTablet for tablet 757ceb24a0024deab1432334dced6de6 (DEFAULT_TABLE table=test-table [id=5dcbecd51b1c4fc7b3732bf9cac6a06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:45.974431 25047 tablet_service.cc:1467] Processing CreateTablet for tablet 757ceb24a0024deab1432334dced6de6 (DEFAULT_TABLE table=test-table [id=5dcbecd51b1c4fc7b3732bf9cac6a06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:45.975122 24784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757ceb24a0024deab1432334dced6de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:45.975317 24916 tablet_service.cc:1467] Processing CreateTablet for tablet 757ceb24a0024deab1432334dced6de6 (DEFAULT_TABLE table=test-table [id=5dcbecd51b1c4fc7b3732bf9cac6a06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:24:45.975845 25047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757ceb24a0024deab1432334dced6de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:45.976400 24916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757ceb24a0024deab1432334dced6de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:24:45.989545 25128 tablet_bootstrap.cc:492] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Bootstrap starting.
I20260620 02:24:45.991983 25129 tablet_bootstrap.cc:492] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Bootstrap starting.
I20260620 02:24:45.994001 25130 tablet_bootstrap.cc:492] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Bootstrap starting.
I20260620 02:24:45.994079 25128 tablet_bootstrap.cc:654] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:45.996232 25129 tablet_bootstrap.cc:654] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:45.996471 25128 log.cc:824] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:45.997591 25129 log.cc:824] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:45.998273 25130 tablet_bootstrap.cc:654] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Neither blocks nor log segments found. Creating new log.
I20260620 02:24:45.999678 25130 log.cc:824] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:46.000070 25128 tablet_bootstrap.cc:492] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: No bootstrap required, opened a new log
I20260620 02:24:46.000427 25128 ts_tablet_manager.cc:1397] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260620 02:24:46.000931 25129 tablet_bootstrap.cc:492] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: No bootstrap required, opened a new log
I20260620 02:24:46.001226 25129 ts_tablet_manager.cc:1397] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:24:46.003322 25130 tablet_bootstrap.cc:492] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: No bootstrap required, opened a new log
I20260620 02:24:46.003625 25130 ts_tablet_manager.cc:1397] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 02:24:46.009656 25129 raft_consensus.cc:348] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.010104 25129 raft_consensus.cc:374] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:46.010222 25129 raft_consensus.cc:729] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3965cb6a0e64b3f9f403e60c38a9dcf, State: Initialized, Role: FOLLOWER
I20260620 02:24:46.010910 25129 consensus_queue.cc:260] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.012390 25128 raft_consensus.cc:348] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.012876 25128 raft_consensus.cc:374] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:46.012964 25128 raft_consensus.cc:729] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906e10f5d1eb4bb1b8acd194c2c0c885, State: Initialized, Role: FOLLOWER
I20260620 02:24:46.013024 25129 ts_tablet_manager.cc:1428] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260620 02:24:46.013711 25128 consensus_queue.cc:260] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.014545 25130 raft_consensus.cc:348] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.014914 24981 heartbeater.cc:503] Master 127.9.231.126:38605 was elected leader, sending a full tablet report...
I20260620 02:24:46.014999 25130 raft_consensus.cc:374] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:24:46.015125 25130 raft_consensus.cc:729] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6a9989ef164361a6cac149f4b14824, State: Initialized, Role: FOLLOWER
I20260620 02:24:46.015802 25128 ts_tablet_manager.cc:1428] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260620 02:24:46.015858 25130 consensus_queue.cc:260] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.017151 25112 heartbeater.cc:503] Master 127.9.231.126:38605 was elected leader, sending a full tablet report...
I20260620 02:24:46.018581 25130 ts_tablet_manager.cc:1428] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 02:24:46.134486 25113 tablet.cc:2369] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:24:46.148068 24982 tablet.cc:2369] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:24:46.160440 24850 tablet.cc:2369] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:24:46.380605 25134 raft_consensus.cc:482] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:24:46.380932 25134 raft_consensus.cc:504] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.382972 25134 leader_election.cc:288] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795), 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817)
I20260620 02:24:46.391052 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.383327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:35775 (local address 127.9.231.67:41795)
0620 02:24:46.383520 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:46.383526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:46.384324 (+ 798us) server_negotiation.cc:407] Connection header received
0620 02:24:46.384560 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:46.384567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:46.384662 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:46.384789 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:46.385637 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.386664 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.388807 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.389745 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.389902 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.389965 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:46.390529 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 02:24:46.390782 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:24:46.391037 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.383265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:41795 (local address 127.9.231.65:35775)
0620 02:24:46.384119 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:46.384141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:46.384374 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:46.384867 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:46.384871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:46.384906 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:46.385366 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:46.385382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.386816 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.386821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:46.388601 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:46.388617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.390082 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.390090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:46.390301 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.390345 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 02:24:46.390461 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 02:24:46.390590 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":1,"thread_start_us":753,"threads_started":1}
I20260620 02:24:46.391325 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.383583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:40021 (local address 127.9.231.66:45817)
0620 02:24:46.383789 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:46.383793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:46.384859 (+ 1066us) server_negotiation.cc:407] Connection header received
0620 02:24:46.385022 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:46.385029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:46.385115 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:46.385229 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:24:46.386089 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.386934 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.389069 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.390127 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.390306 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.390378 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:46.390863 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 02:24:46.391028 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:24:46.391470 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.383736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:45817 (local address 127.9.231.65:40021)
0620 02:24:46.384726 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:46.384746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:24:46.384868 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:24:46.385329 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:24:46.385333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:24:46.385362 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:24:46.385862 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:46.385885 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.387200 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.387204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:46.388836 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:24:46.388851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.390403 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.390408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:24:46.390650 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.390684 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:24:46.390799 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:24:46.390949 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":41,"thread_start_us":437,"threads_started":1}
I20260620 02:24:46.392668 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757ceb24a0024deab1432334dced6de6" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6a9989ef164361a6cac149f4b14824" is_pre_election: true
I20260620 02:24:46.392717 24936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757ceb24a0024deab1432334dced6de6" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" is_pre_election: true
I20260620 02:24:46.393337 25067 raft_consensus.cc:2454] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 0.
I20260620 02:24:46.393357 24936 raft_consensus.cc:2454] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 0.
I20260620 02:24:46.394172 24739 leader_election.cc:302] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 906e10f5d1eb4bb1b8acd194c2c0c885, 9f6a9989ef164361a6cac149f4b14824, f3965cb6a0e64b3f9f403e60c38a9dcf; no voters:
I20260620 02:24:46.394630 25134 raft_consensus.cc:2781] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:24:46.394790 25134 raft_consensus.cc:482] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:24:46.394902 25134 raft_consensus.cc:3037] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:46.397296 25134 raft_consensus.cc:504] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.398068 25134 leader_election.cc:288] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 election: Requested vote from peers 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795), 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817)
I20260620 02:24:46.398682 24936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757ceb24a0024deab1432334dced6de6" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885"
I20260620 02:24:46.398659 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757ceb24a0024deab1432334dced6de6" candidate_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6a9989ef164361a6cac149f4b14824"
I20260620 02:24:46.398909 24936 raft_consensus.cc:3037] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:46.398904 25067 raft_consensus.cc:3037] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:24:46.402397 24936 raft_consensus.cc:2454] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 1.
I20260620 02:24:46.402393 25067 raft_consensus.cc:2454] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3965cb6a0e64b3f9f403e60c38a9dcf in term 1.
I20260620 02:24:46.403007 24740 leader_election.cc:302] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 906e10f5d1eb4bb1b8acd194c2c0c885, 9f6a9989ef164361a6cac149f4b14824, f3965cb6a0e64b3f9f403e60c38a9dcf; no voters:
I20260620 02:24:46.403499 25134 raft_consensus.cc:2781] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:24:46.404058 25134 raft_consensus.cc:686] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 LEADER]: Becoming Leader. State: Replica: f3965cb6a0e64b3f9f403e60c38a9dcf, State: Running, Role: LEADER
I20260620 02:24:46.404685 25134 consensus_queue.cc:237] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } }
I20260620 02:24:46.410482 24662 catalog_manager.cc:5495] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf reported cstate change: term changed from 0 to 1, leader changed from <none> to f3965cb6a0e64b3f9f403e60c38a9dcf (127.9.231.65). New cstate: current_term: 1 leader_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 34959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 } health_report { overall_health: UNKNOWN } } }
I20260620 02:24:46.428085 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.421941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.9.231.65:34959)
0620 02:24:46.422245 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:46.422254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:46.422275 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:24:46.422383 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:46.422387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:46.422470 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:46.422659 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:24:46.423134 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.423983 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.425502 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.425977 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.426139 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.426723 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:24:46.427372 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:24:46.427556 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:46.427682 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 02:24:46.427778 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:24:46.438103 25067 raft_consensus.cc:1261] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Refusing update from remote peer f3965cb6a0e64b3f9f403e60c38a9dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:46.438179 24936 raft_consensus.cc:1261] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Refusing update from remote peer f3965cb6a0e64b3f9f403e60c38a9dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:24:46.439401 25139 consensus_queue.cc:1035] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "906e10f5d1eb4bb1b8acd194c2c0c885" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:46.439867 25134 consensus_queue.cc:1035] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6a9989ef164361a6cac149f4b14824" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:24:46.454813 25142 mvcc.cc:205] Tried to move back new op lower bound from 7298753685238591488 to 7298753685118353408. Current Snapshot: MvccSnapshot[applied={T|T < 7298753685238591488}]
I20260620 02:24:46.465881 25143 mvcc.cc:205] Tried to move back new op lower bound from 7298753685238591488 to 7298753685118353408. Current Snapshot: MvccSnapshot[applied={T|T < 7298753685238591488}]
I20260620 02:24:46.469281 24664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:24:46.469594 24664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:24:46.473722 24664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Sending DeleteTablet for 3 replicas of tablet 757ceb24a0024deab1432334dced6de6
I20260620 02:24:46.475046 24784 tablet_service.cc:1514] Processing DeleteTablet for tablet 757ceb24a0024deab1432334dced6de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:46 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:34746
I20260620 02:24:46.475270 25047 tablet_service.cc:1514] Processing DeleteTablet for tablet 757ceb24a0024deab1432334dced6de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:46 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:36610
I20260620 02:24:46.475526 24916 tablet_service.cc:1514] Processing DeleteTablet for tablet 757ceb24a0024deab1432334dced6de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:24:46 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:35676
I20260620 02:24:46.476939 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24701
I20260620 02:24:46.488780 25153 tablet_replica.cc:330] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: stopping tablet replica
I20260620 02:24:46.489660 25153 raft_consensus.cc:2229] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:24:46.490866 25153 raft_consensus.cc:2258] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:46.491983 25154 tablet_replica.cc:330] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: stopping tablet replica
I20260620 02:24:46.492666 25154 raft_consensus.cc:2229] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:24:46.493595 25152 tablet_replica.cc:330] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: stopping tablet replica
I20260620 02:24:46.494086 25152 raft_consensus.cc:2229] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:24:46.494848 25152 raft_consensus.cc:2258] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:46.495020 25153 ts_tablet_manager.cc:1905] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:46.495646 25154 raft_consensus.cc:2258] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:24:46.498152 25154 ts_tablet_manager.cc:1905] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:46.499271 25152 ts_tablet_manager.cc:1905] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:24:46.505215 25153 ts_tablet_manager.cc:1918] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:46.505437 25153 log.cc:1196] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/757ceb24a0024deab1432334dced6de6
I20260620 02:24:46.506033 25153 ts_tablet_manager.cc:1939] T 757ceb24a0024deab1432334dced6de6 P 9f6a9989ef164361a6cac149f4b14824: Deleting consensus metadata
I20260620 02:24:46.506915 25154 ts_tablet_manager.cc:1918] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:46.507251 25154 log.cc:1196] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/757ceb24a0024deab1432334dced6de6
I20260620 02:24:46.507910 25154 ts_tablet_manager.cc:1939] T 757ceb24a0024deab1432334dced6de6 P 906e10f5d1eb4bb1b8acd194c2c0c885: Deleting consensus metadata
I20260620 02:24:46.508220 24650 catalog_manager.cc:4841] TS 9f6a9989ef164361a6cac149f4b14824 (127.9.231.67:41795): tablet 757ceb24a0024deab1432334dced6de6 (table test-table [id=5dcbecd51b1c4fc7b3732bf9cac6a06b]) successfully deleted
I20260620 02:24:46.509428 25152 ts_tablet_manager.cc:1918] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:24:46.509665 25152 log.cc:1196] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/757ceb24a0024deab1432334dced6de6
I20260620 02:24:46.510186 24651 catalog_manager.cc:4841] TS 906e10f5d1eb4bb1b8acd194c2c0c885 (127.9.231.66:45817): tablet 757ceb24a0024deab1432334dced6de6 (table test-table [id=5dcbecd51b1c4fc7b3732bf9cac6a06b]) successfully deleted
I20260620 02:24:46.511041 25152 ts_tablet_manager.cc:1939] T 757ceb24a0024deab1432334dced6de6 P f3965cb6a0e64b3f9f403e60c38a9dcf: Deleting consensus metadata
I20260620 02:24:46.513125 24650 catalog_manager.cc:4841] TS f3965cb6a0e64b3f9f403e60c38a9dcf (127.9.231.65:34959): tablet 757ceb24a0024deab1432334dced6de6 (table test-table [id=5dcbecd51b1c4fc7b3732bf9cac6a06b]) successfully deleted
I20260620 02:24:46.520670 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.477859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.9.231.65:34959)
0620 02:24:46.478117 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:46.478126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:46.492858 (+ 14732us) server_negotiation.cc:407] Connection header received
0620 02:24:46.493089 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:46.493097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:46.493216 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:46.493347 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:46.497157 (+ 3810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.498406 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.509158 (+ 10752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.509974 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.510161 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.513748 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:46.513760 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:46.513775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:46.513816 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:46.517411 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:46.518209 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:46.518217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:46.518219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:46.518403 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:46.518828 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:46.518832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:46.518834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:46.519367 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:46.519710 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:46.520042 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 02:24:46.520295 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:24:46.521417 24844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:46.626057 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24701
I20260620 02:24:46.646340 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24853
I20260620 02:24:46.662868 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.647303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.9.231.66:45817)
0620 02:24:46.647607 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:46.647612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:46.647630 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:24:46.647690 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:46.647694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:46.647761 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:46.647897 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:46.649589 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.650396 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.652178 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.652642 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.652802 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.656220 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:46.656240 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:46.656258 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:46.656306 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:46.659754 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:46.660600 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:46.660609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:46.660611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:46.660760 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:46.661195 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:46.661200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:46.661202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:46.661608 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:46.661927 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:46.662261 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 02:24:46.662502 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:24:46.663692 24976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:46.762398 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24853
I20260620 02:24:46.782642 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24984
I20260620 02:24:46.797506 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:46.783659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.9.231.67:41795)
0620 02:24:46.783848 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:24:46.783853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:24:46.783881 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:24:46.783941 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:24:46.783946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:24:46.784061 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:24:46.784157 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:24:46.785591 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.786418 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.787879 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:24:46.788298 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:24:46.788406 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:24:46.791275 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:24:46.791291 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:24:46.791310 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:24:46.791352 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:24:46.794281 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:46.795205 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:46.795213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:46.795215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:46.795377 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:24:46.795859 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:24:46.795864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:24:46.795866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:24:46.796278 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:24:46.796525 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:24:46.796976 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 02:24:46.797176 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:24:46.798244 25107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:46.895546 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24984
I20260620 02:24:46.914723 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 24631
I20260620 02:24:46.916219 24694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:24:47.035962 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 24631
I20260620 02:24:47.059523 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38605
--webserver_interface=127.9.231.126
--webserver_port=37619
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:24:47.404786 25160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:47.405120 25160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:47.405283 25160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:47.412969 25160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:24:47.413081 25160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:47.413167 25160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:47.413268 25160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:24:47.413357 25160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:24:47.424418 25160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=37619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:47.425845 25160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:47.427264 25160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:47.439836 25165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:47.439772 25168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:47.439811 25166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:47.440755 25160 server_base.cc:1034] running on GCE node
I20260620 02:24:47.441545 25160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:47.443233 25160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:47.444585 25160 hybrid_clock.cc:648] HybridClock initialized: now 1781922287444496 us; error 78 us; skew 500 ppm
I20260620 02:24:47.444921 25160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:47 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922287, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:47.454936 25160 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:24:47.456846 25160 webserver.cc:460] Webserver started at http://127.9.231.126:37619/ using document root <none> and password file <none>
I20260620 02:24:47.457638 25160 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:47.457787 25160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:47.463958 25160 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:24:47.469282 25175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:47.471029 25160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:24:47.471331 25160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "bb883c61b2f048c9994c27b136d8bb74"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "dca97dac161e7170081f230aaef2506e"
server_key_iv: "f58b122afda701b6c4344978625630c8"
server_key_version: "encryptionkey@0"
I20260620 02:24:47.472388 25160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:47.487200 25160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:47.492257 25160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:47.492594 25160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:47.513475 25160 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38605
I20260620 02:24:47.513484 25226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38605 every 8 connection(s)
I20260620 02:24:47.515241 25160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:24:47.522503 25227 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:24:47.525019 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 25160
I20260620 02:24:47.526088 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:34959
--local_ip_for_outbound_sockets=127.9.231.65
--tserver_master_addrs=127.9.231.126:38605
--webserver_port=44517
--webserver_interface=127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:47.527201 25227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Bootstrap starting.
I20260620 02:24:47.551503 25227 log.cc:824] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Log is configured to *not* fsync() on all Append() calls
I20260620 02:24:47.568651 25227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:24:47.569288 25227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74: Bootstrap complete.
I20260620 02:24:47.579279 25227 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:47.580237 25227 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb883c61b2f048c9994c27b136d8bb74, State: Initialized, Role: FOLLOWER
I20260620 02:24:47.581040 25227 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:47.581298 25227 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:24:47.581415 25227 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:24:47.581629 25227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:24:47.586128 25227 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:47.586686 25227 leader_election.cc:302] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb883c61b2f048c9994c27b136d8bb74; no voters:
I20260620 02:24:47.587440 25227 leader_election.cc:288] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:24:47.587580 25231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:24:47.588704 25231 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [term 3 LEADER]: Becoming Leader. State: Replica: bb883c61b2f048c9994c27b136d8bb74, State: Running, Role: LEADER
I20260620 02:24:47.589251 25231 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } }
I20260620 02:24:47.589740 25227 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:24:47.591907 25232 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bb883c61b2f048c9994c27b136d8bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } } }
I20260620 02:24:47.592283 25232 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:47.592068 25233 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb883c61b2f048c9994c27b136d8bb74. Latest consensus state: current_term: 3 leader_uuid: "bb883c61b2f048c9994c27b136d8bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb883c61b2f048c9994c27b136d8bb74" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38605 } } }
I20260620 02:24:47.592433 25233 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb883c61b2f048c9994c27b136d8bb74 [sys.catalog]: This master's current role is: LEADER
I20260620 02:24:47.602782 25236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:24:47.614225 25236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:24:47.616530 25236 catalog_manager.cc:1259] Loaded cluster ID: 88c110b2c63e4c18ab6a8d76f414d590
I20260620 02:24:47.616673 25236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 02:24:47.619436 25236 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc02d6cd9ad google::LogMessage::Fail() at ??:0
@ 0x7fc02d6d088a google::LogMessage::SendToLog() at ??:0
@ 0x7fc02d6cd4de google::LogMessage::Flush() at ??:0
@ 0x7fc02d6cea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc043104512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc043101976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc043189610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc043189439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc030319345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc03031c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc03031c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc0302f7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc0335d16db start_thread at ??:0
@ 0x7fc02c56761f clone at ??:0
W20260620 02:24:47.953272 25229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:47.953590 25229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:47.953692 25229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:47.961344 25229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:47.961499 25229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:47.961625 25229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:24:47.971940 25229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:34959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=44517
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:47.973278 25229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:47.974653 25229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:47.989261 25268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:47.989220 25269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:47.989200 25271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:47.990171 25229 server_base.cc:1034] running on GCE node
I20260620 02:24:47.990844 25229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:47.991961 25229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:47.993209 25229 hybrid_clock.cc:648] HybridClock initialized: now 1781922287993143 us; error 60 us; skew 500 ppm
I20260620 02:24:47.993626 25229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:47 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922287, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:47.997139 25229 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:24:47.998649 25229 webserver.cc:460] Webserver started at http://127.9.231.65:44517/ using document root <none> and password file <none>
I20260620 02:24:47.999416 25229 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:47.999538 25229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:48.005496 25229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 02:24:48.008574 25278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.010040 25229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:24:48.010286 25229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "f3965cb6a0e64b3f9f403e60c38a9dcf"
format_stamp: "Formatted at 2026-06-20 02:24:41 on dist-test-slave-9vg1"
server_key: "c9993e61e8feb47a68433804af54d6d7"
server_key_iv: "b7f107d9d7ffcb6a035fdb2051e03a4a"
server_key_version: "encryptionkey@0"
I20260620 02:24:48.011508 25229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:48.028793 25229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:48.033635 25229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:48.033947 25229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:48.035220 25229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:48.036700 25229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:48.036788 25229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.036935 25229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:48.037024 25229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.073719 25229 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:34959
I20260620 02:24:48.073884 25390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:34959 every 8 connection(s)
I20260620 02:24:48.075472 25229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:24:48.075965 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 25229
I20260620 02:24:48.076893 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:45817
--local_ip_for_outbound_sockets=127.9.231.66
--tserver_master_addrs=127.9.231.126:38605
--webserver_port=46355
--webserver_interface=127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:48.081408 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:48.079421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:36327)
0620 02:24:48.080264 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:48.080497 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":343,"threads_started":1}
W20260620 02:24:48.082219 25391 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:38605 (0 consecutive failures): Network error: Failed to ping master at 127.9.231.126:38605: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
I20260620 02:24:48.083282 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:48.082809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:36257)
0620 02:24:48.082978 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:48.083140 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:24:48.084203 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:48.083848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:41967)
0620 02:24:48.083996 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:48.084083 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:24:48.084483 25391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:24:48.429715 25394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:48.430055 25394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:48.430222 25394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:48.438122 25394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:48.438267 25394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:48.438477 25394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:24:48.449373 25394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:45817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=46355
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:48.450871 25394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:48.452430 25394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:48.466974 25399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:48.466921 25402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:48.466895 25400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:48.467850 25394 server_base.cc:1034] running on GCE node
I20260620 02:24:48.468784 25394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:48.469836 25394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:48.471195 25394 hybrid_clock.cc:648] HybridClock initialized: now 1781922288471130 us; error 71 us; skew 500 ppm
I20260620 02:24:48.471683 25394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:48 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922288, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:48.475613 25394 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:24:48.477363 25394 webserver.cc:460] Webserver started at http://127.9.231.66:46355/ using document root <none> and password file <none>
I20260620 02:24:48.478140 25394 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:48.478300 25394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:48.484397 25394 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:24:48.487901 25409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.489573 25394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:24:48.489828 25394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "906e10f5d1eb4bb1b8acd194c2c0c885"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "2299bdea03ec680d93dd65343c2e845e"
server_key_iv: "e58a340241737e439ad2d2db8fcb3f7a"
server_key_version: "encryptionkey@0"
I20260620 02:24:48.490950 25394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:48.507433 25394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:48.512753 25394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:48.513082 25394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:48.514223 25394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:48.515854 25394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:48.515985 25394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.516129 25394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:48.516219 25394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.552240 25394 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:45817
I20260620 02:24:48.552337 25521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:45817 every 8 connection(s)
I20260620 02:24:48.553810 25394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:24:48.554538 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 25394
I20260620 02:24:48.555588 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:41795
--local_ip_for_outbound_sockets=127.9.231.67
--tserver_master_addrs=127.9.231.126:38605
--webserver_port=34817
--webserver_interface=127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:24:48.559815 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:48.557639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:49885)
0620 02:24:48.558558 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:48.558821 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":375,"threads_started":1}
W20260620 02:24:48.560642 25522 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:38605 (0 consecutive failures): Network error: Failed to ping master at 127.9.231.126:38605: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
I20260620 02:24:48.561728 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:48.561320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:41417)
0620 02:24:48.561523 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:48.561603 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 02:24:48.562803 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:48.562374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:37137)
0620 02:24:48.562571 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:48.562679 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 02:24:48.563043 25522 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:24:48.914616 25525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:24:48.914963 25525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:24:48.915163 25525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:24:48.922808 25525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:24:48.922936 25525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:24:48.923182 25525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:24:48.933547 25525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:41795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=34817
--tserver_master_addrs=127.9.231.126:38605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:24:48.935087 25525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:24:48.936617 25525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:24:48.952311 25531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:48.952235 25533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:24:48.952394 25530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:24:48.953254 25525 server_base.cc:1034] running on GCE node
I20260620 02:24:48.953961 25525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:24:48.955147 25525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:24:48.956586 25525 hybrid_clock.cc:648] HybridClock initialized: now 1781922288956421 us; error 48 us; skew 500 ppm
I20260620 02:24:48.957024 25525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:24:48 dist-test-slave-9vg1 krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922288, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:24:48.960683 25525 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:24:48.962387 25525 webserver.cc:460] Webserver started at http://127.9.231.67:34817/ using document root <none> and password file <none>
I20260620 02:24:48.963232 25525 fs_manager.cc:359] Metadata directory not provided
I20260620 02:24:48.963388 25525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:24:48.969012 25525 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:24:48.971937 25540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.973435 25525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:24:48.973625 25525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "9f6a9989ef164361a6cac149f4b14824"
format_stamp: "Formatted at 2026-06-20 02:24:42 on dist-test-slave-9vg1"
server_key: "3967071d4e1b00fef949ca9efe706bc2"
server_key_iv: "4061d196e24362ea3c65cba2ca925989"
server_key_version: "encryptionkey@0"
I20260620 02:24:48.974713 25525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:24:48.990226 25525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:24:48.995167 25525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:24:48.995522 25525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:24:48.996614 25525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:24:48.998005 25525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:24:48.998101 25525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:48.998231 25525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:24:48.998309 25525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:24:49.034792 25525 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:41795
I20260620 02:24:49.034895 25652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:41795 every 8 connection(s)
I20260620 02:24:49.036526 25525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:24:49.041810 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:49.039799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:51619)
0620 02:24:49.040614 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:49.040877 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":301,"threads_started":1}
W20260620 02:24:49.042716 25653 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:38605 (0 consecutive failures): Network error: Failed to ping master at 127.9.231.126:38605: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
I20260620 02:24:49.043787 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:49.043368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:33669)
0620 02:24:49.043559 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:49.043655 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:24:49.044651 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 25525
I20260620 02:24:49.044888 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:49.044453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:49913)
0620 02:24:49.044622 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:49.044726 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:24:49.045197 25653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:24:49.086251 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:49.085159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:55911)
0620 02:24:49.085742 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:49.085896 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":289,"threads_started":1}
I20260620 02:24:49.564620 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:49.563771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:44669)
0620 02:24:49.564259 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:49.564380 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":221,"threads_started":1}
I20260620 02:24:50.046777 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:50.045829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:40827)
0620 02:24:50.046345 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:50.046537 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":245,"threads_started":1}
I20260620 02:24:50.088166 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:50.087393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:38255)
0620 02:24:50.087905 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:50.087993 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":238,"threads_started":1}
I20260620 02:24:50.566514 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:50.565700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:35577)
0620 02:24:50.566242 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:50.566334 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":257,"threads_started":1}
I20260620 02:24:51.048655 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:51.047884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:44837)
0620 02:24:51.048397 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:51.048484 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
I20260620 02:24:51.089888 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:51.089114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:46675)
0620 02:24:51.089558 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:51.089643 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260620 02:24:51.568434 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:51.567665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:49223)
0620 02:24:51.568162 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:51.568254 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260620 02:24:52.051043 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:52.050049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:44847)
0620 02:24:52.050737 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:52.050844 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":354,"threads_started":1}
I20260620 02:24:52.091805 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:52.090927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:60891)
0620 02:24:52.091509 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:52.091599 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":325,"threads_started":1}
I20260620 02:24:52.570397 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:52.569638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:49149)
0620 02:24:52.570105 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:52.570219 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":234,"threads_started":1}
I20260620 02:24:53.052829 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:53.052127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:34939)
0620 02:24:53.052553 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:53.052634 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":209,"threads_started":1}
I20260620 02:24:53.093622 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:53.092797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:44915)
0620 02:24:53.093323 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:53.093416 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":259,"threads_started":1}
I20260620 02:24:53.572425 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:53.571595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:54203)
0620 02:24:53.572153 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:53.572248 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
I20260620 02:24:54.054981 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:54.053994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:49727)
0620 02:24:54.054584 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:54.054677 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":355,"threads_started":1}
I20260620 02:24:54.095759 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:54.094793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:38531)
0620 02:24:54.095375 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:54.095534 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":300,"threads_started":1}
I20260620 02:24:54.574219 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:54.573407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:43875)
0620 02:24:54.573912 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:54.574045 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
I20260620 02:24:55.056783 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:55.056030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:37849)
0620 02:24:55.056515 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:55.056611 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260620 02:24:55.097774 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:55.096902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:56223)
0620 02:24:55.097406 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:55.097490 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":236,"threads_started":1}
I20260620 02:24:55.576432 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:55.575398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:40297)
0620 02:24:55.576048 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:55.576195 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":287,"threads_started":1}
I20260620 02:24:56.058697 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:56.057810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:37549)
0620 02:24:56.058383 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:56.058468 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":249,"threads_started":1}
I20260620 02:24:56.099625 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:56.098774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:54083)
0620 02:24:56.099376 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:56.099456 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":396,"threads_started":1}
I20260620 02:24:56.578488 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:56.577472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:58711)
0620 02:24:56.578113 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:56.578242 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":395,"threads_started":1}
I20260620 02:24:57.060465 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:57.059728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:43251)
0620 02:24:57.060202 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:57.060295 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260620 02:24:57.101419 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:57.100600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:56335)
0620 02:24:57.101110 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:57.101239 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":257,"threads_started":1}
I20260620 02:24:57.580577 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:57.579769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:58995)
0620 02:24:57.580295 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:57.580389 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":253,"threads_started":1}
I20260620 02:24:58.062521 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:58.061535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:54355)
0620 02:24:58.062196 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:58.062291 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":364,"threads_started":1}
I20260620 02:24:58.103405 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:58.102592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:49299)
0620 02:24:58.103049 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:58.103238 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":247,"threads_started":1}
I20260620 02:24:58.582540 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:58.581757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:51577)
0620 02:24:58.582274 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:58.582367 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260620 02:24:59.064443 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:59.063617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:42811)
0620 02:24:59.064163 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:59.064277 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":287,"threads_started":1}
I20260620 02:24:59.105269 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:59.104453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:41215)
0620 02:24:59.104964 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:59.105081 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":284,"threads_started":1}
I20260620 02:24:59.584465 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:24:59.583708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:60763)
0620 02:24:59.584175 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 02:24:59.584280 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260620 02:25:00.066186 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:00.065429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:43847)
0620 02:25:00.065890 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:00.065985 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260620 02:25:00.107352 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:00.106414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:59637)
0620 02:25:00.107014 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:00.107181 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":331,"threads_started":1}
I20260620 02:25:00.586493 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:00.585694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:44787)
0620 02:25:00.586209 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:00.586313 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
I20260620 02:25:01.068277 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:01.067316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:54685)
0620 02:25:01.067912 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:01.068065 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":358,"threads_started":1}
I20260620 02:25:01.109275 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:01.108376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:60805)
0620 02:25:01.108965 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:01.109072 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":286,"threads_started":1}
I20260620 02:25:01.588430 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:01.587601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:55717)
0620 02:25:01.588114 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:01.588236 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":252,"threads_started":1}
I20260620 02:25:02.070389 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:02.069465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:54625)
0620 02:25:02.070046 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:02.070179 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":287,"threads_started":1}
I20260620 02:25:02.111197 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:02.110336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:56467)
0620 02:25:02.110825 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:02.110915 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260620 02:25:02.590267 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:02.589428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:35157)
0620 02:25:02.589956 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:02.590067 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":256,"threads_started":1}
I20260620 02:25:03.072347 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:03.071591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:51615)
0620 02:25:03.072063 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:03.072157 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":240,"threads_started":1}
I20260620 02:25:03.113242 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:03.112404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:51021)
0620 02:25:03.112918 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:03.113021 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":290,"threads_started":1}
I20260620 02:25:03.592547 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:03.591776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.66:45683)
0620 02:25:03.592258 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:03.592358 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":234,"threads_started":1}
I20260620 02:25:04.058483 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 25229
I20260620 02:25:04.074885 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:04.073985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.67:49685)
0620 02:25:04.074558 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:04.074680 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":314,"threads_started":1}
I20260620 02:25:04.115507 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:04.114437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38605 (local address 127.9.231.65:33091)
0620 02:25:04.115118 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:04.115301 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":351,"threads_started":1}
I20260620 02:25:04.344177 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:04.060022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.9.231.65:34959)
0620 02:25:04.060739 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:04.060758 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:04.323996 (+263238us) server_negotiation.cc:407] Connection header received
0620 02:25:04.326874 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:04.326889 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:04.327601 (+ 712us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:04.327947 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:04.329481 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:04.330813 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:04.332602 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:04.333409 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:04.333568 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:04.337043 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:04.337053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:04.337071 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:04.337120 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:04.340285 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:04.341095 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:04.341102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:04.341109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:04.341312 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:04.341821 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:04.341824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:04.341826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:04.342420 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:04.342670 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:04.343369 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 02:25:04.343861 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":322,"threads_started":1}
I20260620 02:25:04.345664 25386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:04.434414 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 25229
I20260620 02:25:04.453912 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 25394
I20260620 02:25:04.474084 25705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:04.455246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.9.231.66:45817)
0620 02:25:04.455757 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:04.455772 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:04.455811 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:25:04.457407 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:04.457420 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:04.457892 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:04.458307 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:04.460143 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:04.461595 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:04.463214 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:04.463993 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:04.464225 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:04.467013 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:04.467021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:04.467039 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:04.467152 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:04.470348 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:04.471164 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:04.471172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:04.471180 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:04.471442 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:04.471874 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:04.471879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:04.471883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:04.472468 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:04.472808 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:04.473282 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 02:25:04.473768 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":221,"threads_started":1}
I20260620 02:25:04.475740 25517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:04.564254 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 25394
I20260620 02:25:04.584085 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 25525
I20260620 02:25:04.602874 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:04.585107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.9.231.67:41795)
0620 02:25:04.585655 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:04.585669 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:04.585707 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:25:04.587393 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:04.587403 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:04.587848 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:04.588220 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:04.589775 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:04.591052 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:04.592499 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:04.593217 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:04.593437 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:04.596232 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:04.596240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:04.596257 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:04.596296 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:04.599407 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:04.600177 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:04.600184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:04.600192 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:04.600370 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:04.600779 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:04.600784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:04.600787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:04.601339 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:04.601670 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:04.602136 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 02:25:04.602567 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
I20260620 02:25:04.604420 25648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:04.692581 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 25525
2026-06-20T02:25:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25459 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:25:04 dist-test-slave-9vg1 krb5kdc[25711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:25:04 dist-test-slave-9vg1 krb5kdc[25711](info): set up 2 sockets
Jun 20 02:25:04 dist-test-slave-9vg1 krb5kdc[25711](info): commencing operation
krb5kdc: starting...
W20260620 02:25:05.840982 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:25:06 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:25:06.140162 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33803
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:25:06.504226 25724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:06.504580 25724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:06.504799 25724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:06.512471 25724 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:25:06.512606 25724 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:25:06.512698 25724 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:25:06.512789 25724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:25:06.512893 25724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:06.513015 25724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:06.513104 25724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:25:06.513227 25724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:25:06.524076 25724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:06.525590 25724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:06.527148 25724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:06.542032 25729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:06.541950 25732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:06.541991 25730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:06.543355 25724 server_base.cc:1034] running on GCE node
I20260620 02:25:06.544375 25724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:06.546159 25724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:06.547477 25724 hybrid_clock.cc:648] HybridClock initialized: now 1781922306547405 us; error 44 us; skew 500 ppm
I20260620 02:25:06.547914 25724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:06 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922306, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:06.552038 25724 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:25:06.553791 25724 webserver.cc:460] Webserver started at http://127.9.231.126:41275/ using document root <none> and password file <none>
I20260620 02:25:06.554607 25724 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:06.554754 25724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:06.555171 25724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:25:06.558449 25724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "0da91db920e446e78249e22aed2fcd3c"
format_stamp: "Formatted at 2026-06-20 02:25:06 on dist-test-slave-9vg1"
server_key: "573a0e83790c3cb6a5036169dc738d38"
server_key_iv: "1e727a82e89488f7ef8ed2884777713b"
server_key_version: "encryptionkey@0"
I20260620 02:25:06.559458 25724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "0da91db920e446e78249e22aed2fcd3c"
format_stamp: "Formatted at 2026-06-20 02:25:06 on dist-test-slave-9vg1"
server_key: "573a0e83790c3cb6a5036169dc738d38"
server_key_iv: "1e727a82e89488f7ef8ed2884777713b"
server_key_version: "encryptionkey@0"
I20260620 02:25:06.565672 25724 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:25:06.570191 25739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:06.571868 25724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:25:06.572093 25724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "0da91db920e446e78249e22aed2fcd3c"
format_stamp: "Formatted at 2026-06-20 02:25:06 on dist-test-slave-9vg1"
server_key: "573a0e83790c3cb6a5036169dc738d38"
server_key_iv: "1e727a82e89488f7ef8ed2884777713b"
server_key_version: "encryptionkey@0"
I20260620 02:25:06.572412 25724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:06.599218 25724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:06.603802 25724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:06.604120 25724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:06.624971 25724 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33803
I20260620 02:25:06.625028 25790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33803 every 8 connection(s)
I20260620 02:25:06.626425 25724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:25:06.627347 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 25724
I20260620 02:25:06.627709 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:25:06.628506 10141 external_mini_cluster.cc:1428] Setting key 7d1024a95326169c8f294b43f659a712
I20260620 02:25:06.631453 25791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:25:06 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:25:06.646965 25791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Bootstrap starting.
I20260620 02:25:06.651937 25791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Neither blocks nor log segments found. Creating new log.
I20260620 02:25:06.653316 25791 log.cc:824] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:06.657285 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:06.630617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.9.231.126:33803)
0620 02:25:06.631598 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:06.631617 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:06.631679 (+ 62us) server_negotiation.cc:407] Connection header received
0620 02:25:06.633418 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:06.633444 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:06.633876 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:06.634312 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:06.636319 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:06.637955 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:06.640084 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:06.640873 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:06.641028 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:06.647016 (+ 5988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:06.647029 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:06.647054 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:06.647142 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:06.651521 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:06.652557 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:06.652566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:06.652575 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:06.652807 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:06.653346 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:06.653352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:06.653355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:06.654154 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:06.654395 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:06.655304 (+ 909us) server_negotiation.cc:299] Negotiation successful
0620 02:25:06.655784 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":468,"threads_started":1}
I20260620 02:25:06.657536 25791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: No bootstrap required, opened a new log
I20260620 02:25:06.666246 25791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:06.666774 25791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:25:06.666944 25791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da91db920e446e78249e22aed2fcd3c, State: Initialized, Role: FOLLOWER
I20260620 02:25:06.667665 25791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:06.667940 25791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:25:06.668097 25791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:25:06.668282 25791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:25:06.670578 25791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:06.671257 25791 leader_election.cc:302] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da91db920e446e78249e22aed2fcd3c; no voters:
I20260620 02:25:06.672075 25791 leader_election.cc:288] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:25:06.672242 25796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:25:06.673133 25796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 LEADER]: Becoming Leader. State: Replica: 0da91db920e446e78249e22aed2fcd3c, State: Running, Role: LEADER
I20260620 02:25:06.673892 25796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:06.674475 25791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:25:06.677299 25797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0da91db920e446e78249e22aed2fcd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } } }
I20260620 02:25:06.677259 25798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da91db920e446e78249e22aed2fcd3c. Latest consensus state: current_term: 1 leader_uuid: "0da91db920e446e78249e22aed2fcd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } } }
I20260620 02:25:06.678016 25797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:25:06.678013 25798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:25:06.679415 25803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:25:06.688643 25803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:25:06.698894 25803 catalog_manager.cc:1347] Generated new cluster ID: f82897b736de45278969cfcfe2cbc7d8
I20260620 02:25:06.699030 25803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:25:06.729130 25803 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:25:06.730571 25803 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:25:06.740150 25803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Generated new TSK 0
I20260620 02:25:06.741158 25803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:25:06.819633 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33803
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:25:07.196754 25817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:07.197067 25817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:07.197268 25817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:07.204741 25817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:07.204861 25817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:07.205051 25817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:25:07.214985 25817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:07.216415 25817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:07.217788 25817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:07.232273 25823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:07.232328 25825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:07.232259 25822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:07.233047 25817 server_base.cc:1034] running on GCE node
I20260620 02:25:07.233920 25817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:07.235016 25817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:07.236400 25817 hybrid_clock.cc:648] HybridClock initialized: now 1781922307236346 us; error 35 us; skew 500 ppm
I20260620 02:25:07.236835 25817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:07 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922307, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:07.240899 25817 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:25:07.242745 25817 webserver.cc:460] Webserver started at http://127.9.231.65:33551/ using document root <none> and password file <none>
I20260620 02:25:07.243640 25817 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:07.243757 25817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:07.244084 25817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:25:07.247414 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "db72f0aea2464a0e98600f9676166591"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "65de3fd291e60a80ce4ba01e814e9600"
server_key_iv: "04c840f5cb9a2e4487f694974671d06b"
server_key_version: "encryptionkey@0"
I20260620 02:25:07.248250 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "db72f0aea2464a0e98600f9676166591"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "65de3fd291e60a80ce4ba01e814e9600"
server_key_iv: "04c840f5cb9a2e4487f694974671d06b"
server_key_version: "encryptionkey@0"
I20260620 02:25:07.254233 25817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:25:07.258370 25832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:07.260476 25817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:25:07.260772 25817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "db72f0aea2464a0e98600f9676166591"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "65de3fd291e60a80ce4ba01e814e9600"
server_key_iv: "04c840f5cb9a2e4487f694974671d06b"
server_key_version: "encryptionkey@0"
I20260620 02:25:07.261037 25817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:07.297314 25817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:07.302608 25817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:07.302958 25817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:07.304301 25817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:07.305884 25817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:07.305967 25817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:07.306164 25817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:07.306245 25817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:07.344861 25817 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:42577
I20260620 02:25:07.344928 25944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:42577 every 8 connection(s)
I20260620 02:25:07.346324 25817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:25:07.351739 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 25817
I20260620 02:25:07.351970 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:25:07.352623 10141 external_mini_cluster.cc:1428] Setting key 4ff415f8bbcc20aae4618a34ab64bc2a
Jun 20 02:25:07 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922307, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:25:07.369349 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:07.349118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:49315 (local address 127.9.231.126:33803)
0620 02:25:07.349611 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:07.349618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:07.350738 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 02:25:07.352975 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:07.352981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:07.353102 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:07.353238 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:07.355200 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.356316 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.357987 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.358436 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.358589 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:07.362406 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:07.362416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:07.362423 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:07.362476 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:07.365358 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:07.366535 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:07.366540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:07.366542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:07.366701 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:07.367332 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:07.367337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:07.367339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:07.367686 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:07.367818 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:07.368841 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0620 02:25:07.369046 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":228,"threads_started":1}
I20260620 02:25:07.370891 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:07.349735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:49315)
0620 02:25:07.350548 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:07.350595 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:07.352578 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:07.353528 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:07.353542 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:07.354192 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:25:07.354983 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:07.355005 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.356495 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.356498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:07.357820 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:07.357833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.358720 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.358727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:07.358996 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:07.359852 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:25:07.359877 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:25:07.362122 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:25:07.365511 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:07.365525 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:07.365551 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:07.366320 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:07.366842 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:07.366850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:07.366857 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:07.367171 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:07.367829 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:25:07.367840 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:25:07.368371 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 02:25:07.368710 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 02:25:07.369338 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":381,"threads_started":1}
I20260620 02:25:07.372725 25945 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33803
I20260620 02:25:07.373056 25945 heartbeater.cc:464] Registering TS with master...
I20260620 02:25:07.373842 25945 heartbeater.cc:511] Master 127.9.231.126:33803 requested a full tablet report, sending...
I20260620 02:25:07.376495 25756 ts_manager.cc:194] Registered new tserver with Master: db72f0aea2464a0e98600f9676166591 (127.9.231.65:42577)
I20260620 02:25:07.378175 25756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:49315
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:25:07.412619 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33803
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:25:07.769186 25951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:07.769536 25951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:07.769721 25951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:07.777709 25951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:07.777872 25951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:07.778056 25951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:25:07.788694 25951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:07.790174 25951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:07.791652 25951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:07.805933 25956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:07.805968 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:07.806010 25957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:07.807024 25951 server_base.cc:1034] running on GCE node
I20260620 02:25:07.807801 25951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:07.808889 25951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:07.810168 25951 hybrid_clock.cc:648] HybridClock initialized: now 1781922307810115 us; error 56 us; skew 500 ppm
I20260620 02:25:07.810545 25951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:07 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922307, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:07.813882 25951 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:25:07.815546 25951 webserver.cc:460] Webserver started at http://127.9.231.66:41761/ using document root <none> and password file <none>
I20260620 02:25:07.816277 25951 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:07.816430 25951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:07.816756 25951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:25:07.820025 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "f4803b65224c4ab7a3f537507a974404"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "a6fa261f1ca79eb3aefe2c10db0086a4"
server_key_iv: "7317ab94c946b15dfc97952b3262183f"
server_key_version: "encryptionkey@0"
I20260620 02:25:07.820879 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "f4803b65224c4ab7a3f537507a974404"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "a6fa261f1ca79eb3aefe2c10db0086a4"
server_key_iv: "7317ab94c946b15dfc97952b3262183f"
server_key_version: "encryptionkey@0"
I20260620 02:25:07.826475 25951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:25:07.830765 25966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:07.832388 25951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:25:07.832619 25951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "f4803b65224c4ab7a3f537507a974404"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "a6fa261f1ca79eb3aefe2c10db0086a4"
server_key_iv: "7317ab94c946b15dfc97952b3262183f"
server_key_version: "encryptionkey@0"
I20260620 02:25:07.832860 25951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:07.858193 25951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:07.862900 25951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:07.863330 25951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:07.864411 25951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:07.866153 25951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:07.866254 25951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:07.866410 25951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:07.866508 25951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:07.904150 25951 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:34775
I20260620 02:25:07.904253 26078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:34775 every 8 connection(s)
I20260620 02:25:07.905630 25951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:25:07.910117 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 25951
I20260620 02:25:07.910324 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:25:07.910943 10141 external_mini_cluster.cc:1428] Setting key 8cd00c35368db49984d4063af12aac8e
Jun 20 02:25:07 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922307, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:25:07.932037 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:07.908400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:51437 (local address 127.9.231.126:33803)
0620 02:25:07.908909 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:07.908913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:07.909913 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 02:25:07.912430 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:07.912435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:07.912537 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:07.912676 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:07.914979 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.915874 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.918565 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.919129 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.919259 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:07.923960 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:07.923970 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:07.923976 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:07.924029 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:07.928012 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:07.928992 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:07.929001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:07.929007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:07.929182 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:07.929698 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:07.929706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:07.929712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:07.930131 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:07.930291 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:07.931235 (+ 944us) server_negotiation.cc:299] Negotiation successful
0620 02:25:07.931521 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":285,"threads_started":1}
I20260620 02:25:07.932864 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:07.908859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:51437)
0620 02:25:07.909613 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:07.909675 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:07.912152 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:07.913004 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:07.913022 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:07.913840 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:25:07.914757 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:07.914787 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.916085 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.916094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:07.918376 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:07.918395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:07.919401 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:07.919411 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:07.919774 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:07.920596 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:25:07.920633 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:25:07.923632 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:25:07.928215 (+ 4583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:07.928224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:07.928241 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:07.928830 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:07.929308 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:07.929311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:07.929314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:07.929571 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:07.930308 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:25:07.930315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:25:07.930749 (+ 434us) client_negotiation.cc:769] Sending connection context
0620 02:25:07.931025 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 02:25:07.931567 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":275,"threads_started":1}
I20260620 02:25:07.934579 26079 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33803
I20260620 02:25:07.934931 26079 heartbeater.cc:464] Registering TS with master...
I20260620 02:25:07.935734 26079 heartbeater.cc:511] Master 127.9.231.126:33803 requested a full tablet report, sending...
I20260620 02:25:07.937520 25756 ts_manager.cc:194] Registered new tserver with Master: f4803b65224c4ab7a3f537507a974404 (127.9.231.66:34775)
I20260620 02:25:07.938524 25756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:51437
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:25:07.974593 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33803
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:25:08.329588 26085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:08.329965 26085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:08.330148 26085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:08.338063 26085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:08.338177 26085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:08.338292 26085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:25:08.351779 26085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:08.353008 26085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:08.354272 26085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:08.368011 26090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:08.367957 26093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:08.367923 26091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:08.368873 26085 server_base.cc:1034] running on GCE node
I20260620 02:25:08.369622 26085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:08.370751 26085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:08.371992 26085 hybrid_clock.cc:648] HybridClock initialized: now 1781922308371935 us; error 50 us; skew 500 ppm
I20260620 02:25:08.372332 26085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:08 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922308, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:08.375872 26085 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:25:08.377655 26085 webserver.cc:460] Webserver started at http://127.9.231.67:46245/ using document root <none> and password file <none>
I20260620 02:25:08.378374 26085 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:08.378515 26085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:08.378854 26085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:25:08.381884 25945 heartbeater.cc:503] Master 127.9.231.126:33803 was elected leader, sending a full tablet report...
I20260620 02:25:08.382294 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "d693bcee1b1d4dd18d25717c1d3e694f"
format_stamp: "Formatted at 2026-06-20 02:25:08 on dist-test-slave-9vg1"
server_key: "e61d21b4aa6bc878055aa9bac073eeac"
server_key_iv: "dd916a2c7642bbb7f8d009336715c783"
server_key_version: "encryptionkey@0"
I20260620 02:25:08.383322 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "d693bcee1b1d4dd18d25717c1d3e694f"
format_stamp: "Formatted at 2026-06-20 02:25:08 on dist-test-slave-9vg1"
server_key: "e61d21b4aa6bc878055aa9bac073eeac"
server_key_iv: "dd916a2c7642bbb7f8d009336715c783"
server_key_version: "encryptionkey@0"
I20260620 02:25:08.389288 26085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:25:08.393348 26100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:08.395038 26085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:25:08.395319 26085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "d693bcee1b1d4dd18d25717c1d3e694f"
format_stamp: "Formatted at 2026-06-20 02:25:08 on dist-test-slave-9vg1"
server_key: "e61d21b4aa6bc878055aa9bac073eeac"
server_key_iv: "dd916a2c7642bbb7f8d009336715c783"
server_key_version: "encryptionkey@0"
I20260620 02:25:08.395550 26085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:08.413827 26085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:08.418679 26085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:08.419001 26085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:08.420073 26085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:08.421603 26085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:08.421684 26085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:08.421775 26085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:08.421819 26085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:08.460028 26085 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:34801
I20260620 02:25:08.460068 26212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:34801 every 8 connection(s)
I20260620 02:25:08.461520 26085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:25:08.471935 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 26085
I20260620 02:25:08.472163 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:25:08.472694 10141 external_mini_cluster.cc:1428] Setting key cc370b9e8041e2522f708390ea59c486
Jun 20 02:25:08 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922308, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:25:08.484323 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.464388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:42925 (local address 127.9.231.126:33803)
0620 02:25:08.464893 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.464898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.466022 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 02:25:08.468181 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.468187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.468307 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.468433 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:08.470175 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.470936 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.472738 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.473208 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.473347 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.477762 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:08.477767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:08.477771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:08.477807 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:08.481033 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.481982 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.481987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.481989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.482180 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.482601 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.482604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.482606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.482896 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:08.483038 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.483833 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.484036 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":249,"threads_started":1}
I20260620 02:25:08.485343 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.465026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:42925)
0620 02:25:08.465855 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:08.465899 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:08.467872 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:08.468702 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:08.468711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:08.469352 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:25:08.469975 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.469995 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.471141 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.471152 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.472581 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.472595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.473472 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.473480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.473765 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.474625 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:25:08.474657 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:25:08.477520 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:25:08.481190 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:08.481198 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:08.481214 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:08.481802 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:08.482294 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:08.482297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:08.482300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:08.482498 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:08.483003 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:25:08.483009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:25:08.483438 (+ 429us) client_negotiation.cc:769] Sending connection context
0620 02:25:08.483683 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 02:25:08.484094 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":318,"threads_started":1}
I20260620 02:25:08.486893 26213 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33803
I20260620 02:25:08.487291 26213 heartbeater.cc:464] Registering TS with master...
I20260620 02:25:08.488224 26213 heartbeater.cc:511] Master 127.9.231.126:33803 requested a full tablet report, sending...
I20260620 02:25:08.490031 25756 ts_manager.cc:194] Registered new tserver with Master: d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801)
I20260620 02:25:08.490936 25756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:42925
I20260620 02:25:08.501981 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:25:08.520944 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.507206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.9.231.126:33803)
0620 02:25:08.507397 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.507401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.507980 (+ 579us) server_negotiation.cc:407] Connection header received
0620 02:25:08.508167 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.508171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.508240 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.508333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:08.509912 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.510711 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.512379 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.512802 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.512900 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.515601 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:08.515606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:08.515609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:08.515657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:08.518223 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.519023 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.519027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.519029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.519254 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.519624 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.519627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.519629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.519969 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:08.520163 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.520479 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.520709 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:25:08.525748 25756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:25:08.533372 25756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:25:08.560720 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.549303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.9.231.67:34801)
0620 02:25:08.550359 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.550371 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.550977 (+ 606us) server_negotiation.cc:407] Connection header received
0620 02:25:08.551295 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.551304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.551596 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.553078 (+ 1482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:08.553423 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.554778 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.557739 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.559252 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.559392 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.559643 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.560148 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.560320 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":684,"threads_started":1}
I20260620 02:25:08.560801 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.548159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34801 (local address 127.0.0.1:48816)
0620 02:25:08.550853 (+ 2694us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:08.550868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:08.551024 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:08.552147 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:08.552158 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:08.552205 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:08.552814 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.552830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.554939 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.554945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.557541 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.557556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.559781 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.559786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.559970 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.559993 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:25:08.560106 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:25:08.560321 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2582,"mutex_wait_us":2,"thread_start_us":3611,"threads_started":1}
I20260620 02:25:08.561429 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.547598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:34775 (local address 127.0.0.1:59520)
0620 02:25:08.548799 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:08.548858 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:08.549081 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:08.554539 (+ 5458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:08.554543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:08.554569 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:08.554987 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.555002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.557030 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.557038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.558622 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.558635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.560614 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.560621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.560793 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.560828 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:25:08.560914 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:25:08.561052 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":642,"threads_started":1}
I20260620 02:25:08.562054 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.549841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.9.231.65:42577)
0620 02:25:08.551609 (+ 1768us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.551618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.552317 (+ 699us) server_negotiation.cc:407] Connection header received
0620 02:25:08.552420 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.552427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.552693 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.552879 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:08.553562 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.555124 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.557667 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.559241 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.559397 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.559639 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.561524 (+ 1885us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.561738 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1585,"thread_start_us":1414,"threads_started":1}
I20260620 02:25:08.562170 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.547996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.9.231.66:34775)
0620 02:25:08.553936 (+ 5940us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.553948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.553977 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:25:08.554075 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.554092 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.554314 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.554518 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:08.555204 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.556873 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.558791 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.560128 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.560267 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.560455 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.561707 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.561878 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5781,"thread_start_us":5688,"threads_started":1}
I20260620 02:25:08.562541 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.548859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:42577 (local address 127.0.0.1:51306)
0620 02:25:08.552064 (+ 3205us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:08.552084 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:08.552195 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:08.553069 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:08.553075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:08.553111 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:08.553393 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.553401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.555319 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.555326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.557490 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.557505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.559447 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.559453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.559682 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.559739 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 02:25:08.561238 (+ 1499us) client_negotiation.cc:240] Negotiation successful
0620 02:25:08.561376 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3074,"mutex_wait_us":31,"thread_start_us":1750,"threads_started":1}
I20260620 02:25:08.565505 26014 tablet_service.cc:1467] Processing CreateTablet for tablet a322eee8898642729a3cdbf856f651b9 (DEFAULT_TABLE table=test-table [id=3164b2577f7a45139d684ba421fb24f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:25:08.565503 26148 tablet_service.cc:1467] Processing CreateTablet for tablet a322eee8898642729a3cdbf856f651b9 (DEFAULT_TABLE table=test-table [id=3164b2577f7a45139d684ba421fb24f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:25:08.566135 25880 tablet_service.cc:1467] Processing CreateTablet for tablet a322eee8898642729a3cdbf856f651b9 (DEFAULT_TABLE table=test-table [id=3164b2577f7a45139d684ba421fb24f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:25:08.567034 26014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a322eee8898642729a3cdbf856f651b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:25:08.567114 26148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a322eee8898642729a3cdbf856f651b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:25:08.567476 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a322eee8898642729a3cdbf856f651b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:25:08.582425 26234 tablet_bootstrap.cc:492] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Bootstrap starting.
I20260620 02:25:08.585371 26232 tablet_bootstrap.cc:492] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Bootstrap starting.
I20260620 02:25:08.586759 26234 tablet_bootstrap.cc:654] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Neither blocks nor log segments found. Creating new log.
I20260620 02:25:08.587898 26233 tablet_bootstrap.cc:492] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Bootstrap starting.
I20260620 02:25:08.588399 26234 log.cc:824] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:08.589735 26232 tablet_bootstrap.cc:654] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Neither blocks nor log segments found. Creating new log.
I20260620 02:25:08.590965 26232 log.cc:824] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:08.592556 26234 tablet_bootstrap.cc:492] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: No bootstrap required, opened a new log
I20260620 02:25:08.592767 26233 tablet_bootstrap.cc:654] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Neither blocks nor log segments found. Creating new log.
I20260620 02:25:08.592857 26234 ts_tablet_manager.cc:1397] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 02:25:08.593700 26232 tablet_bootstrap.cc:492] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: No bootstrap required, opened a new log
I20260620 02:25:08.593966 26232 ts_tablet_manager.cc:1397] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 02:25:08.594259 26233 log.cc:824] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:08.597690 26233 tablet_bootstrap.cc:492] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: No bootstrap required, opened a new log
I20260620 02:25:08.597983 26233 ts_tablet_manager.cc:1397] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:25:08.602954 26232 raft_consensus.cc:348] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.603483 26232 raft_consensus.cc:374] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:25:08.603602 26232 raft_consensus.cc:729] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4803b65224c4ab7a3f537507a974404, State: Initialized, Role: FOLLOWER
I20260620 02:25:08.604326 26232 consensus_queue.cc:260] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.604338 26234 raft_consensus.cc:348] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.604851 26234 raft_consensus.cc:374] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:25:08.604990 26234 raft_consensus.cc:729] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d693bcee1b1d4dd18d25717c1d3e694f, State: Initialized, Role: FOLLOWER
I20260620 02:25:08.605608 26079 heartbeater.cc:503] Master 127.9.231.126:33803 was elected leader, sending a full tablet report...
I20260620 02:25:08.605736 26234 consensus_queue.cc:260] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.606251 26232 ts_tablet_manager.cc:1428] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 02:25:08.610016 26233 raft_consensus.cc:348] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.610706 26233 raft_consensus.cc:374] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:25:08.610834 26233 raft_consensus.cc:729] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db72f0aea2464a0e98600f9676166591, State: Initialized, Role: FOLLOWER
I20260620 02:25:08.611640 26233 consensus_queue.cc:260] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.612188 26213 heartbeater.cc:503] Master 127.9.231.126:33803 was elected leader, sending a full tablet report...
I20260620 02:25:08.612607 26234 ts_tablet_manager.cc:1428] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Time spent starting tablet: real 0.020s user 0.014s sys 0.006s
I20260620 02:25:08.613343 26233 ts_tablet_manager.cc:1428] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260620 02:25:08.657809 26080 tablet.cc:2369] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:25:08.662916 26238 raft_consensus.cc:482] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:25:08.663244 26238 raft_consensus.cc:504] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.665040 26238 leader_election.cc:288] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801), db72f0aea2464a0e98600f9676166591 (127.9.231.65:42577)
I20260620 02:25:08.675874 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.665482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:55113 (local address 127.9.231.67:34801)
0620 02:25:08.665757 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.665763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.666976 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 02:25:08.667259 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.667265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.667356 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.667418 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:08.668486 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.669695 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.673139 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.674485 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.674640 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.674723 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.675454 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.675604 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 02:25:08.675884 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.665643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:47797 (local address 127.9.231.65:42577)
0620 02:25:08.665864 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.665870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.666975 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 02:25:08.667270 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.667275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.667350 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.667418 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:08.668487 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.669695 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.673135 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.674451 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.674640 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.674723 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.675454 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.675604 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:25:08.676887 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.665386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34801 (local address 127.9.231.66:55113)
0620 02:25:08.666863 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:08.666880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:08.667034 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:08.667633 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:08.667639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:08.667689 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:08.668286 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.668306 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.669951 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.669958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.672905 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.672925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.674936 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.674948 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.675198 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.675280 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 02:25:08.676091 (+ 811us) client_negotiation.cc:240] Negotiation successful
0620 02:25:08.676292 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1373,"mutex_wait_us":236,"spinlock_wait_cycles":13696,"thread_start_us":1002,"threads_started":1}
I20260620 02:25:08.676909 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.665333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:42577 (local address 127.9.231.66:47797)
0620 02:25:08.666824 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:08.666852 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:08.667106 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:08.667633 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:08.667639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:08.667689 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:08.668310 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.668319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.669951 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.669958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.672905 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:08.672925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.674953 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.674958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:08.675203 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.675284 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 02:25:08.676091 (+ 807us) client_negotiation.cc:240] Negotiation successful
0620 02:25:08.676292 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":397,"spinlock_wait_cycles":44672,"thread_start_us":926,"threads_started":1}
I20260620 02:25:08.678115 26168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a322eee8898642729a3cdbf856f651b9" candidate_uuid: "f4803b65224c4ab7a3f537507a974404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" is_pre_election: true
I20260620 02:25:08.678597 26168 raft_consensus.cc:2454] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4803b65224c4ab7a3f537507a974404 in term 0.
I20260620 02:25:08.679325 25968 leader_election.cc:302] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d693bcee1b1d4dd18d25717c1d3e694f, f4803b65224c4ab7a3f537507a974404; no voters:
I20260620 02:25:08.679448 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a322eee8898642729a3cdbf856f651b9" candidate_uuid: "f4803b65224c4ab7a3f537507a974404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72f0aea2464a0e98600f9676166591" is_pre_election: true
I20260620 02:25:08.679831 26238 raft_consensus.cc:2781] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:25:08.680001 26238 raft_consensus.cc:482] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:25:08.680116 26238 raft_consensus.cc:3037] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:25:08.680061 25900 raft_consensus.cc:2454] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4803b65224c4ab7a3f537507a974404 in term 0.
I20260620 02:25:08.682723 26238 raft_consensus.cc:504] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.683418 26238 leader_election.cc:288] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [CANDIDATE]: Term 1 election: Requested vote from peers d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801), db72f0aea2464a0e98600f9676166591 (127.9.231.65:42577)
I20260620 02:25:08.684181 26168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a322eee8898642729a3cdbf856f651b9" candidate_uuid: "f4803b65224c4ab7a3f537507a974404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d693bcee1b1d4dd18d25717c1d3e694f"
I20260620 02:25:08.684425 26168 raft_consensus.cc:3037] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:25:08.684522 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a322eee8898642729a3cdbf856f651b9" candidate_uuid: "f4803b65224c4ab7a3f537507a974404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72f0aea2464a0e98600f9676166591"
I20260620 02:25:08.684760 25900 raft_consensus.cc:3037] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:25:08.687885 26168 raft_consensus.cc:2454] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4803b65224c4ab7a3f537507a974404 in term 1.
I20260620 02:25:08.688004 25900 raft_consensus.cc:2454] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4803b65224c4ab7a3f537507a974404 in term 1.
I20260620 02:25:08.688421 25968 leader_election.cc:302] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d693bcee1b1d4dd18d25717c1d3e694f, f4803b65224c4ab7a3f537507a974404; no voters:
I20260620 02:25:08.688788 26238 raft_consensus.cc:2781] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:25:08.689206 26238 raft_consensus.cc:686] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 1 LEADER]: Becoming Leader. State: Replica: f4803b65224c4ab7a3f537507a974404, State: Running, Role: LEADER
I20260620 02:25:08.689893 26238 consensus_queue.cc:237] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } }
I20260620 02:25:08.694729 25755 catalog_manager.cc:5495] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4803b65224c4ab7a3f537507a974404 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "f4803b65224c4ab7a3f537507a974404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } health_report { overall_health: UNKNOWN } } }
W20260620 02:25:08.713542 26214 tablet.cc:2369] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:25:08.718714 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.712900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.9.231.66:34775)
0620 02:25:08.713094 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.713103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.713121 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:25:08.713344 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.713349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.713463 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.713654 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:25:08.714051 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.714967 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.716600 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.717014 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.717118 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.717697 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:25:08.718224 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:25:08.718338 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.718413 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.718484 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:25:08.727487 26168 raft_consensus.cc:1261] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Refusing update from remote peer f4803b65224c4ab7a3f537507a974404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:25:08.727629 25900 raft_consensus.cc:1261] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 1 FOLLOWER]: Refusing update from remote peer f4803b65224c4ab7a3f537507a974404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:25:08.728662 26243 consensus_queue.cc:1035] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:25:08.729050 26238 consensus_queue.cc:1035] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:25:08.745883 26248 mvcc.cc:205] Tried to move back new op lower bound from 7298753776536203264 to 7298753776397869056. Current Snapshot: MvccSnapshot[applied={T|T < 7298753776536203264}]
I20260620 02:25:08.756978 25755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:25:08.757308 25755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:25:08.761513 25755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Sending DeleteTablet for 3 replicas of tablet a322eee8898642729a3cdbf856f651b9
I20260620 02:25:08.763005 26014 tablet_service.cc:1514] Processing DeleteTablet for tablet a322eee8898642729a3cdbf856f651b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:25:08 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:59520
I20260620 02:25:08.763362 26148 tablet_service.cc:1514] Processing DeleteTablet for tablet a322eee8898642729a3cdbf856f651b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:25:08 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:48816
I20260620 02:25:08.763868 25880 tablet_service.cc:1514] Processing DeleteTablet for tablet a322eee8898642729a3cdbf856f651b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:25:08 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:51306
I20260620 02:25:08.764391 26257 tablet_replica.cc:330] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: stopping tablet replica
I20260620 02:25:08.764545 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 25817
I20260620 02:25:08.764976 26257 raft_consensus.cc:2229] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:25:08.765890 26257 raft_consensus.cc:2258] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:25:08.769073 26257 ts_tablet_manager.cc:1905] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:25:08.776429 26258 tablet_replica.cc:330] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: stopping tablet replica
I20260620 02:25:08.778707 26258 raft_consensus.cc:2229] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:25:08.779515 26258 raft_consensus.cc:2258] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:25:08.779614 26257 ts_tablet_manager.cc:1918] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:25:08.779850 26257 log.cc:1196] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/a322eee8898642729a3cdbf856f651b9
I20260620 02:25:08.780653 26257 ts_tablet_manager.cc:1939] T a322eee8898642729a3cdbf856f651b9 P d693bcee1b1d4dd18d25717c1d3e694f: Deleting consensus metadata
I20260620 02:25:08.780995 26256 tablet_replica.cc:330] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: stopping tablet replica
I20260620 02:25:08.781481 26256 raft_consensus.cc:2229] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:25:08.782156 26256 raft_consensus.cc:2258] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:25:08.782270 26258 ts_tablet_manager.cc:1905] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:25:08.783144 25741 catalog_manager.cc:4841] TS d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801): tablet a322eee8898642729a3cdbf856f651b9 (table test-table [id=3164b2577f7a45139d684ba421fb24f6]) successfully deleted
Jun 20 02:25:08 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:25:08.786489 26256 ts_tablet_manager.cc:1905] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:25:08.790122 26258 ts_tablet_manager.cc:1918] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:25:08.790373 26258 log.cc:1196] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/a322eee8898642729a3cdbf856f651b9
I20260620 02:25:08.790987 26258 ts_tablet_manager.cc:1939] T a322eee8898642729a3cdbf856f651b9 P db72f0aea2464a0e98600f9676166591: Deleting consensus metadata
I20260620 02:25:08.793359 25742 catalog_manager.cc:4841] TS db72f0aea2464a0e98600f9676166591 (127.9.231.65:42577): tablet a322eee8898642729a3cdbf856f651b9 (table test-table [id=3164b2577f7a45139d684ba421fb24f6]) successfully deleted
I20260620 02:25:08.794466 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.765766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.9.231.65:42577)
0620 02:25:08.765951 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.765957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.771661 (+ 5704us) server_negotiation.cc:407] Connection header received
0620 02:25:08.771914 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.771920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.772017 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.772152 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:08.775741 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.777002 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.778641 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.779337 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.779466 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.785869 (+ 6403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:08.785885 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:08.785904 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:08.785963 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:08.790230 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.791591 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.791603 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.791606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.791870 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.792381 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.792386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.792390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.792876 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:08.793186 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.793745 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.794004 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:25:08.795027 26256 ts_tablet_manager.cc:1918] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:25:08.795280 25940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:08.795300 26256 log.cc:1196] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/a322eee8898642729a3cdbf856f651b9
I20260620 02:25:08.795863 26256 ts_tablet_manager.cc:1939] T a322eee8898642729a3cdbf856f651b9 P f4803b65224c4ab7a3f537507a974404: Deleting consensus metadata
I20260620 02:25:08.797864 25740 catalog_manager.cc:4841] TS f4803b65224c4ab7a3f537507a974404 (127.9.231.66:34775): tablet a322eee8898642729a3cdbf856f651b9 (table test-table [id=3164b2577f7a45139d684ba421fb24f6]) successfully deleted
I20260620 02:25:08.893442 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 25817
I20260620 02:25:08.920540 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 25951
Jun 20 02:25:08 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:25:08.941458 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:08.921359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.9.231.66:34775)
0620 02:25:08.921557 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:08.921564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:08.921850 (+ 286us) server_negotiation.cc:407] Connection header received
0620 02:25:08.922040 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:08.922045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:08.922138 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:08.922258 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:08.924019 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.925374 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.926998 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:08.927786 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:08.927931 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:08.932915 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:08.932939 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:08.932963 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:08.933030 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:08.937653 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.938613 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.938629 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.938634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.938841 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:08.939428 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:08.939436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:08.939439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:08.940003 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:08.940484 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:08.940812 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 02:25:08.941062 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:25:08.942239 26074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:09.062574 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 25951
I20260620 02:25:09.085351 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 26085
Jun 20 02:25:09 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:25:09.102099 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:09.086119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.9.231.67:34801)
0620 02:25:09.086364 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:09.086368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:09.086387 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:25:09.086465 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:09.086469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:09.086532 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:09.086610 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:09.087997 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:09.088795 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:09.090157 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:09.090569 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:09.090668 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:09.095634 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:09.095652 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:09.095671 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:09.095726 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:09.098581 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:09.099734 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:09.099747 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:09.099751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:09.099937 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:09.100390 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:09.100400 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:09.100402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:09.100802 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:09.101124 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:09.101464 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 02:25:09.101716 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 02:25:09.102887 26208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:09.196193 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 26085
I20260620 02:25:09.214852 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 25724
I20260620 02:25:09.216212 25786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:09.334442 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 25724
I20260620 02:25:29.358590 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33803
--webserver_interface=127.9.231.126
--webserver_port=41275
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:25:29.712426 26264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:29.712757 26264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:29.712922 26264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:29.720571 26264 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:25:29.720697 26264 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:25:29.720773 26264 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:25:29.720851 26264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:25:29.720947 26264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:29.721084 26264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:29.721169 26264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:25:29.721252 26264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:25:29.731786 26264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=41275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:29.733228 26264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:29.734640 26264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:29.760021 26271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:29.760046 26270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:29.759987 26273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:29.761137 26264 server_base.cc:1034] running on GCE node
I20260620 02:25:29.762166 26264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:29.764086 26264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:29.765390 26264 hybrid_clock.cc:648] HybridClock initialized: now 1781922329765309 us; error 61 us; skew 500 ppm
I20260620 02:25:29.765750 26264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:29 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922329, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:29.776926 26264 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:25:29.778882 26264 webserver.cc:460] Webserver started at http://127.9.231.126:41275/ using document root <none> and password file <none>
I20260620 02:25:29.779834 26264 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:29.779956 26264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:29.786988 26264 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 02:25:29.790673 26280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:29.792361 26264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:25:29.792598 26264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "0da91db920e446e78249e22aed2fcd3c"
format_stamp: "Formatted at 2026-06-20 02:25:06 on dist-test-slave-9vg1"
server_key: "573a0e83790c3cb6a5036169dc738d38"
server_key_iv: "1e727a82e89488f7ef8ed2884777713b"
server_key_version: "encryptionkey@0"
I20260620 02:25:29.793808 26264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:29.807899 26264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:29.813102 26264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:29.813465 26264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:29.834571 26264 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33803
I20260620 02:25:29.834614 26331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33803 every 8 connection(s)
I20260620 02:25:29.836376 26264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:25:29.844194 26332 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:25:29.846513 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 26264
I20260620 02:25:29.847532 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:42577
--local_ip_for_outbound_sockets=127.9.231.65
--tserver_master_addrs=127.9.231.126:33803
--webserver_port=33551
--webserver_interface=127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:25:29.848042 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Bootstrap starting.
I20260620 02:25:29.869468 26332 log.cc:824] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:29.882880 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:25:29.883569 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Bootstrap complete.
I20260620 02:25:29.893462 26332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:29.894409 26332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da91db920e446e78249e22aed2fcd3c, State: Initialized, Role: FOLLOWER
I20260620 02:25:29.895213 26332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:29.895486 26332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:25:29.895627 26332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:25:29.895831 26332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:25:29.900362 26332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:29.900930 26332 leader_election.cc:302] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da91db920e446e78249e22aed2fcd3c; no voters:
I20260620 02:25:29.901793 26332 leader_election.cc:288] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:25:29.901942 26336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:25:29.903395 26336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 LEADER]: Becoming Leader. State: Replica: 0da91db920e446e78249e22aed2fcd3c, State: Running, Role: LEADER
I20260620 02:25:29.904042 26336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:29.904600 26332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:25:29.906955 26337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0da91db920e446e78249e22aed2fcd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } } }
I20260620 02:25:29.907549 26337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:25:29.911468 26338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da91db920e446e78249e22aed2fcd3c. Latest consensus state: current_term: 2 leader_uuid: "0da91db920e446e78249e22aed2fcd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } } }
I20260620 02:25:29.912037 26338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:25:29.913993 26341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:25:29.924584 26341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:25:29.926779 26341 catalog_manager.cc:1259] Loaded cluster ID: f82897b736de45278969cfcfe2cbc7d8
I20260620 02:25:29.927009 26341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:25:29.932639 26341 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:25:29.935340 26341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Loaded TSK: 0
I20260620 02:25:29.976889 26341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Generated new TSK 1
I20260620 02:25:29.977375 26341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:25:30.238636 26334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:30.239043 26334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:30.239317 26334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:30.247440 26334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:30.247641 26334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:30.247833 26334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:25:30.258785 26334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:42577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=33551
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:30.260318 26334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:30.261893 26334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:30.277966 26359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:30.277977 26360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:30.278906 26362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:30.279371 26334 server_base.cc:1034] running on GCE node
I20260620 02:25:30.280083 26334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:30.281193 26334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:30.282498 26334 hybrid_clock.cc:648] HybridClock initialized: now 1781922330282407 us; error 50 us; skew 500 ppm
I20260620 02:25:30.282868 26334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:30 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922330, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:30.286582 26334 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:25:30.288396 26334 webserver.cc:460] Webserver started at http://127.9.231.65:33551/ using document root <none> and password file <none>
I20260620 02:25:30.289134 26334 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:30.289302 26334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:30.295377 26334 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:25:30.298671 26369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:30.300303 26334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:25:30.300500 26334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "db72f0aea2464a0e98600f9676166591"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "65de3fd291e60a80ce4ba01e814e9600"
server_key_iv: "04c840f5cb9a2e4487f694974671d06b"
server_key_version: "encryptionkey@0"
I20260620 02:25:30.301539 26334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:30.321516 26334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:30.326154 26334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:30.326468 26334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:30.327639 26334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:30.329116 26334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:30.329216 26334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:30.329377 26334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:30.329484 26334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:30.365291 26334 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:42577
I20260620 02:25:30.365348 26481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:42577 every 8 connection(s)
I20260620 02:25:30.366905 26334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:25:30.376924 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 26334
I20260620 02:25:30.377777 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:34775
--local_ip_for_outbound_sockets=127.9.231.66
--tserver_master_addrs=127.9.231.126:33803
--webserver_port=41761
--webserver_interface=127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:25:30 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922330, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:25:30.395218 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:30.370124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:55649 (local address 127.9.231.126:33803)
0620 02:25:30.370802 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:30.370814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:30.371601 (+ 787us) server_negotiation.cc:407] Connection header received
0620 02:25:30.373893 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:30.373907 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:30.374359 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:30.374728 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:30.376811 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.378990 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.380964 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.382154 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.382387 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:30.386544 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:30.386554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:30.386573 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:30.386634 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:30.390239 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:30.391166 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:30.391174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:30.391179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:30.391357 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:30.391844 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:30.391849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:30.391851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:30.392356 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:30.392573 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:30.393401 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 02:25:30.393797 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":261,"threads_started":1}
I20260620 02:25:30.395218 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:30.370443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:55649)
0620 02:25:30.371380 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:30.371435 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:30.373244 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:30.374933 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:30.374941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:30.375830 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:25:30.376490 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:30.376512 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.379263 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.379267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:30.380768 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:30.380786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.382441 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.382447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:30.382663 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:30.383513 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:25:30.383543 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:25:30.386187 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:25:30.390436 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:30.390447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:30.390464 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:30.390952 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:30.391488 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:30.391493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:30.391497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:30.391708 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:30.392518 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:25:30.392527 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:25:30.392902 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 02:25:30.393180 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 02:25:30.393797 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":374,"threads_started":1}
I20260620 02:25:30.398864 26482 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33803
I20260620 02:25:30.399240 26482 heartbeater.cc:464] Registering TS with master...
I20260620 02:25:30.400046 26482 heartbeater.cc:511] Master 127.9.231.126:33803 requested a full tablet report, sending...
I20260620 02:25:30.402904 26297 ts_manager.cc:194] Registered new tserver with Master: db72f0aea2464a0e98600f9676166591 (127.9.231.65:42577)
I20260620 02:25:30.405211 26297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:55649
W20260620 02:25:30.724306 26486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:30.724627 26486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:30.724725 26486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:30.732035 26486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:30.732156 26486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:30.732268 26486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:25:30.742177 26486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:34775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=41761
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:30.743556 26486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:30.744815 26486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:30.758338 26492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:30.758311 26494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:30.758255 26491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:30.759320 26486 server_base.cc:1034] running on GCE node
I20260620 02:25:30.760063 26486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:30.761168 26486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:30.762420 26486 hybrid_clock.cc:648] HybridClock initialized: now 1781922330762362 us; error 46 us; skew 500 ppm
I20260620 02:25:30.762852 26486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:30 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922330, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:30.766796 26486 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:25:30.768639 26486 webserver.cc:460] Webserver started at http://127.9.231.66:41761/ using document root <none> and password file <none>
I20260620 02:25:30.769448 26486 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:30.769645 26486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:30.776187 26486 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:25:30.779314 26501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:30.780952 26486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:25:30.781186 26486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "f4803b65224c4ab7a3f537507a974404"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "a6fa261f1ca79eb3aefe2c10db0086a4"
server_key_iv: "7317ab94c946b15dfc97952b3262183f"
server_key_version: "encryptionkey@0"
I20260620 02:25:30.782337 26486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:30.814880 26486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:30.819952 26486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:30.820297 26486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:30.821444 26486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:30.822937 26486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:30.823026 26486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:30.823220 26486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:30.823314 26486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:30.861914 26486 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:34775
I20260620 02:25:30.861984 26613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:34775 every 8 connection(s)
I20260620 02:25:30.863984 26486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:25:30.865634 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 26486
I20260620 02:25:30.866504 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:34801
--local_ip_for_outbound_sockets=127.9.231.67
--tserver_master_addrs=127.9.231.126:33803
--webserver_port=46245
--webserver_interface=127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:25:30 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922330, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:25:30.889355 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:30.867700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:46087 (local address 127.9.231.126:33803)
0620 02:25:30.867954 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:30.867963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:30.868949 (+ 986us) server_negotiation.cc:407] Connection header received
0620 02:25:30.870858 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:30.870868 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:30.870987 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:30.871161 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:30.872788 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.873918 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.876626 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.877061 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.877189 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:30.882069 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:30.882075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:30.882078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:30.882116 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:30.884949 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:30.886097 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:30.886102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:30.886105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:30.886477 (+ 372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:30.887141 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:30.887148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:30.887151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:30.887601 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:30.887773 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:30.888827 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0620 02:25:30.889070 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:25:30.891037 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:30.867894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:46087)
0620 02:25:30.868739 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:30.868784 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:30.870554 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:30.871430 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:30.871443 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:30.872008 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:25:30.872590 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:30.872609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.874089 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.874104 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:30.876441 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:30.876459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:30.877359 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:30.877369 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:30.877712 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:30.878549 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:25:30.878583 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:25:30.881732 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:25:30.885127 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:30.885140 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:30.885164 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:30.885863 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:30.886619 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:30.886627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:30.886633 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:30.886939 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:30.887750 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:25:30.887763 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:25:30.888329 (+ 566us) client_negotiation.cc:769] Sending connection context
0620 02:25:30.888690 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 02:25:30.889375 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":352,"threads_started":1}
I20260620 02:25:30.892804 26614 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33803
I20260620 02:25:30.893155 26614 heartbeater.cc:464] Registering TS with master...
I20260620 02:25:30.893929 26614 heartbeater.cc:511] Master 127.9.231.126:33803 requested a full tablet report, sending...
I20260620 02:25:30.895543 26297 ts_manager.cc:194] Registered new tserver with Master: f4803b65224c4ab7a3f537507a974404 (127.9.231.66:34775)
I20260620 02:25:30.896417 26297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:46087
W20260620 02:25:31.223642 26617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:31.223938 26617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:31.224099 26617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:31.231422 26617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:31.231540 26617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:31.231743 26617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:25:31.244956 26617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:34801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=46245
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:31.246410 26617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:31.247807 26617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:31.261232 26623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:31.261368 26625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:31.261992 26622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:31.262454 26617 server_base.cc:1034] running on GCE node
I20260620 02:25:31.263110 26617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:31.264242 26617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:31.265476 26617 hybrid_clock.cc:648] HybridClock initialized: now 1781922331265406 us; error 59 us; skew 500 ppm
I20260620 02:25:31.265822 26617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:31 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922331, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:31.269375 26617 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:25:31.271029 26617 webserver.cc:460] Webserver started at http://127.9.231.67:46245/ using document root <none> and password file <none>
I20260620 02:25:31.271814 26617 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:31.271965 26617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:31.281072 26617 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 02:25:31.284786 26632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:31.286527 26617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:25:31.286720 26617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "d693bcee1b1d4dd18d25717c1d3e694f"
format_stamp: "Formatted at 2026-06-20 02:25:08 on dist-test-slave-9vg1"
server_key: "e61d21b4aa6bc878055aa9bac073eeac"
server_key_iv: "dd916a2c7642bbb7f8d009336715c783"
server_key_version: "encryptionkey@0"
I20260620 02:25:31.288337 26617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:31.322070 26617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:31.328298 26617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:31.328652 26617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:31.330039 26617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:31.332091 26617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:31.332221 26617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:31.332332 26617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:31.332417 26617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:31.368016 26617 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:34801
I20260620 02:25:31.368134 26744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:34801 every 8 connection(s)
I20260620 02:25:31.369884 26617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:25:31.376883 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 26617
Jun 20 02:25:31 dist-test-slave-9vg1 krb5kdc[25711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922331, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:25:31.395553 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.378133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.9.231.126:33803)
0620 02:25:31.378902 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.378907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.378922 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:25:31.378966 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.378969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.379020 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.379171 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:31.380976 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.382067 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.383921 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.384349 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.384498 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.388239 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:31.388246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:31.388250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:31.388298 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:31.391971 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:31.392891 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:31.392897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:31.392900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:31.393052 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:31.393606 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:31.393611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:31.393645 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:31.393949 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:31.394232 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.394707 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.395020 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":547,"threads_started":1}
I20260620 02:25:31.395658 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.372935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:58659 (local address 127.9.231.126:33803)
0620 02:25:31.373178 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.373185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.374391 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 02:25:31.376448 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.376453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.376529 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.376670 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:31.378810 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.380031 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.382248 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.382991 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.383171 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.387573 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:31.387579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:31.387582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:31.387625 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:31.391273 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:31.392419 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:31.392426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:31.392430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:31.392575 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:31.393170 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:31.393175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:31.393178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:31.393582 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:31.393745 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.394847 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.395252 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":94,"spinlock_wait_cycles":7040}
I20260620 02:25:31.396744 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.373415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:58659)
0620 02:25:31.374200 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:31.374247 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:31.376066 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:31.376949 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:31.376963 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:31.377734 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:25:31.378566 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.378596 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.380212 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.380217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.382065 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.382080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.383292 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.383297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.383506 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.384283 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:25:31.384312 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:25:31.387284 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:25:31.391470 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:31.391484 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:31.391517 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:31.392284 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:31.392705 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:25:31.392708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:25:31.392711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:25:31.392999 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:25:31.393817 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:25:31.393824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:25:31.394277 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 02:25:31.394658 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 02:25:31.395270 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":323,"threads_started":1}
I20260620 02:25:31.398468 26745 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33803
I20260620 02:25:31.398820 26745 heartbeater.cc:464] Registering TS with master...
I20260620 02:25:31.399712 26745 heartbeater.cc:511] Master 127.9.231.126:33803 requested a full tablet report, sending...
I20260620 02:25:31.401134 26297 ts_manager.cc:194] Registered new tserver with Master: d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801)
I20260620 02:25:31.401983 26297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:58659
I20260620 02:25:31.409441 26482 heartbeater.cc:503] Master 127.9.231.126:33803 was elected leader, sending a full tablet report...
I20260620 02:25:31.412175 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:25:31.419744 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.413081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.9.231.126:33803)
0620 02:25:31.413270 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.413275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.413934 (+ 659us) server_negotiation.cc:407] Connection header received
0620 02:25:31.414125 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.414129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.414305 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.414436 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:25:31.415188 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.415934 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.417741 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.418192 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.418287 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.418879 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:25:31.419220 (+ 341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 02:25:31.419539 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55116: token expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:25:31.419902 26749 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55116: token expired
W20260620 02:25:31.420598 26219 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.231.126:33803)
I20260620 02:25:31.433898 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.421087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.9.231.126:33803)
0620 02:25:31.421259 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.421263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.421294 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:25:31.421437 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.421441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.421506 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.421582 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:31.422946 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.423771 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.425280 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.425691 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.425785 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.428527 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:31.428532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:31.428535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:31.428571 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:31.431337 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:31.432049 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:31.432052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:31.432055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:31.432218 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:31.432589 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:31.432593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:31.432595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:31.432938 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:31.433129 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.433460 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.433634 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:25:31.437951 26297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:25:31.445731 26297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:25:31.487663 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.460473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.9.231.67:34801)
0620 02:25:31.476462 (+ 15989us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.476478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.477759 (+ 1281us) server_negotiation.cc:407] Connection header received
0620 02:25:31.477917 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.477930 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.478307 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.478571 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:31.480255 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.481897 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.484683 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.486241 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.486394 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.486902 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.487081 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.487264 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15703,"thread_start_us":16016,"threads_started":1}
I20260620 02:25:31.488008 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.459876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34801 (local address 127.0.0.1:46392)
0620 02:25:31.477333 (+ 17457us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:31.477361 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:31.477568 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:31.479469 (+ 1901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:31.479478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:31.479513 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:31.479960 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.479976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.482082 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.482088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.484448 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.484460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.486466 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.486473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.486691 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.486755 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 02:25:31.486882 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 02:25:31.487261 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17268,"mutex_wait_us":56,"thread_start_us":17092,"threads_started":1}
I20260620 02:25:31.488161 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.459790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:42577 (local address 127.0.0.1:37022)
0620 02:25:31.472397 (+ 12607us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:31.472425 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:31.472611 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:31.480282 (+ 7671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:31.480289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:31.480324 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:31.480737 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.480746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.482347 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.482355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.484867 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.484882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.487115 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.487122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.487296 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.487332 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:25:31.487439 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 02:25:31.487633 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12424,"mutex_wait_us":41,"thread_start_us":12250,"threads_started":1}
I20260620 02:25:31.488576 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.459471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:34775 (local address 127.0.0.1:48854)
0620 02:25:31.471849 (+ 12378us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:31.471910 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:31.472239 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:31.479958 (+ 7719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:31.479963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:31.479991 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:31.480490 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.480503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.483424 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.483435 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.485663 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.485677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.487674 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.487681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.487870 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.487906 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:25:31.487970 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 02:25:31.488118 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11553,"mutex_wait_us":3,"thread_start_us":11382,"threads_started":1}
I20260620 02:25:31.489212 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.459558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.9.231.65:42577)
0620 02:25:31.479631 (+ 20073us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.479640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.479671 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:25:31.479767 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.479774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.480010 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.480264 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:31.480900 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.482217 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.485053 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.486529 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.486680 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.486925 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.488570 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.488877 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19884,"thread_start_us":19745,"threads_started":1}
I20260620 02:25:31.490494 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.460062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.9.231.66:34775)
0620 02:25:31.479419 (+ 19357us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.479428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.479455 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:25:31.479551 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.479559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.479750 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.479958 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:31.481146 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.483272 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.485859 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.487406 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.488780 (+ 1374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.488980 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.489047 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.489291 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19177,"thread_start_us":19031,"threads_started":1}
I20260620 02:25:31.492604 26680 tablet_service.cc:1467] Processing CreateTablet for tablet 79b259cb872f4b5eae54c77054d4f3fb (DEFAULT_TABLE table=test-table [id=3b76c6aaaa524c8aa468a7d9a21615cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:25:31.492663 26417 tablet_service.cc:1467] Processing CreateTablet for tablet 79b259cb872f4b5eae54c77054d4f3fb (DEFAULT_TABLE table=test-table [id=3b76c6aaaa524c8aa468a7d9a21615cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:25:31.493409 26549 tablet_service.cc:1467] Processing CreateTablet for tablet 79b259cb872f4b5eae54c77054d4f3fb (DEFAULT_TABLE table=test-table [id=3b76c6aaaa524c8aa468a7d9a21615cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:25:31.494206 26417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b259cb872f4b5eae54c77054d4f3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:25:31.494206 26680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b259cb872f4b5eae54c77054d4f3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:25:31.494630 26549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b259cb872f4b5eae54c77054d4f3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:25:31.508365 26761 tablet_bootstrap.cc:492] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Bootstrap starting.
I20260620 02:25:31.511860 26763 tablet_bootstrap.cc:492] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Bootstrap starting.
I20260620 02:25:31.513500 26761 tablet_bootstrap.cc:654] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Neither blocks nor log segments found. Creating new log.
I20260620 02:25:31.514894 26761 log.cc:824] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:31.515769 26763 tablet_bootstrap.cc:654] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Neither blocks nor log segments found. Creating new log.
I20260620 02:25:31.516199 26762 tablet_bootstrap.cc:492] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Bootstrap starting.
I20260620 02:25:31.516920 26763 log.cc:824] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:31.518383 26761 tablet_bootstrap.cc:492] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: No bootstrap required, opened a new log
I20260620 02:25:31.519451 26761 ts_tablet_manager.cc:1397] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 02:25:31.519726 26762 tablet_bootstrap.cc:654] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Neither blocks nor log segments found. Creating new log.
I20260620 02:25:31.520990 26762 log.cc:824] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:31.521119 26763 tablet_bootstrap.cc:492] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: No bootstrap required, opened a new log
I20260620 02:25:31.521404 26763 ts_tablet_manager.cc:1397] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:25:31.524238 26762 tablet_bootstrap.cc:492] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: No bootstrap required, opened a new log
I20260620 02:25:31.524505 26762 ts_tablet_manager.cc:1397] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 02:25:31.527446 26761 raft_consensus.cc:348] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.527850 26761 raft_consensus.cc:374] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:25:31.527945 26761 raft_consensus.cc:729] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4803b65224c4ab7a3f537507a974404, State: Initialized, Role: FOLLOWER
I20260620 02:25:31.528641 26761 consensus_queue.cc:260] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.530778 26614 heartbeater.cc:503] Master 127.9.231.126:33803 was elected leader, sending a full tablet report...
I20260620 02:25:31.531402 26761 ts_tablet_manager.cc:1428] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260620 02:25:31.533586 26763 raft_consensus.cc:348] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.534060 26763 raft_consensus.cc:374] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:25:31.534168 26763 raft_consensus.cc:729] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db72f0aea2464a0e98600f9676166591, State: Initialized, Role: FOLLOWER
I20260620 02:25:31.534915 26762 raft_consensus.cc:348] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.535372 26762 raft_consensus.cc:374] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:25:31.535498 26762 raft_consensus.cc:729] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d693bcee1b1d4dd18d25717c1d3e694f, State: Initialized, Role: FOLLOWER
I20260620 02:25:31.535642 26763 consensus_queue.cc:260] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.536103 26762 consensus_queue.cc:260] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.537869 26763 ts_tablet_manager.cc:1428] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 02:25:31.540345 26762 ts_tablet_manager.cc:1428] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20260620 02:25:31.540421 26745 heartbeater.cc:503] Master 127.9.231.126:33803 was elected leader, sending a full tablet report...
W20260620 02:25:31.618288 26615 tablet.cc:2369] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:25:31.619784 26483 tablet.cc:2369] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:25:31.621727 26746 tablet.cc:2369] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:25:31.847103 26768 raft_consensus.cc:482] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:25:31.847437 26768 raft_consensus.cc:504] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.849273 26768 leader_election.cc:288] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4803b65224c4ab7a3f537507a974404 (127.9.231.66:34775), d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801)
I20260620 02:25:31.856365 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.849776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:36687 (local address 127.9.231.66:34775)
0620 02:25:31.850004 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.850009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.850765 (+ 756us) server_negotiation.cc:407] Connection header received
0620 02:25:31.851005 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.851013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.851151 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.851255 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:31.851814 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.852553 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.854490 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.855348 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.855558 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.855643 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.855908 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.856089 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:25:31.856437 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.849886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:34775 (local address 127.9.231.65:36687)
0620 02:25:31.850631 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:31.850650 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:31.850844 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:31.851278 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:31.851282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:31.851320 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:31.851667 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.851675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.852699 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.852706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.854292 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.854305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.855574 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.855580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.855741 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.855765 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:25:31.855871 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:25:31.856027 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":1,"thread_start_us":395,"threads_started":1}
I20260620 02:25:31.857661 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.849868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:37209 (local address 127.9.231.67:34801)
0620 02:25:31.850089 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.850113 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.851308 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 02:25:31.851531 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.851537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.851614 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.851725 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:25:31.852292 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.853210 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.855428 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.856550 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.856713 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.856785 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.857355 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.857468 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:25:31.857625 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b259cb872f4b5eae54c77054d4f3fb" candidate_uuid: "db72f0aea2464a0e98600f9676166591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4803b65224c4ab7a3f537507a974404" is_pre_election: true
I20260620 02:25:31.857998 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.849886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:34801 (local address 127.9.231.65:37209)
0620 02:25:31.851186 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:31.851202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:25:31.851369 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:25:31.851743 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:25:31.851748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:25:31.851774 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:25:31.852102 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.852111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.853352 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.853359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.855241 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:25:31.855255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.856917 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.856922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:25:31.857150 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.857189 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:25:31.857309 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:25:31.857460 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":243,"thread_start_us":654,"threads_started":1}
I20260620 02:25:31.858321 26569 raft_consensus.cc:2454] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db72f0aea2464a0e98600f9676166591 in term 0.
I20260620 02:25:31.859242 26370 leader_election.cc:302] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db72f0aea2464a0e98600f9676166591, f4803b65224c4ab7a3f537507a974404; no voters:
I20260620 02:25:31.859268 26700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b259cb872f4b5eae54c77054d4f3fb" candidate_uuid: "db72f0aea2464a0e98600f9676166591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" is_pre_election: true
I20260620 02:25:31.859759 26768 raft_consensus.cc:2781] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:25:31.859808 26700 raft_consensus.cc:2454] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db72f0aea2464a0e98600f9676166591 in term 0.
I20260620 02:25:31.859933 26768 raft_consensus.cc:482] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:25:31.860033 26768 raft_consensus.cc:3037] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:25:31.863394 26768 raft_consensus.cc:504] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.864192 26768 leader_election.cc:288] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [CANDIDATE]: Term 1 election: Requested vote from peers f4803b65224c4ab7a3f537507a974404 (127.9.231.66:34775), d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801)
I20260620 02:25:31.864784 26700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b259cb872f4b5eae54c77054d4f3fb" candidate_uuid: "db72f0aea2464a0e98600f9676166591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d693bcee1b1d4dd18d25717c1d3e694f"
I20260620 02:25:31.864842 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b259cb872f4b5eae54c77054d4f3fb" candidate_uuid: "db72f0aea2464a0e98600f9676166591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4803b65224c4ab7a3f537507a974404"
I20260620 02:25:31.865054 26569 raft_consensus.cc:3037] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:25:31.865048 26700 raft_consensus.cc:3037] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:25:31.868580 26700 raft_consensus.cc:2454] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db72f0aea2464a0e98600f9676166591 in term 1.
I20260620 02:25:31.868587 26569 raft_consensus.cc:2454] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db72f0aea2464a0e98600f9676166591 in term 1.
I20260620 02:25:31.869143 26370 leader_election.cc:302] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db72f0aea2464a0e98600f9676166591, f4803b65224c4ab7a3f537507a974404; no voters:
I20260620 02:25:31.869729 26768 raft_consensus.cc:2781] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:25:31.870261 26768 raft_consensus.cc:686] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 1 LEADER]: Becoming Leader. State: Replica: db72f0aea2464a0e98600f9676166591, State: Running, Role: LEADER
I20260620 02:25:31.870890 26768 consensus_queue.cc:237] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } }
I20260620 02:25:31.876390 26297 catalog_manager.cc:5495] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 reported cstate change: term changed from 0 to 1, leader changed from <none> to db72f0aea2464a0e98600f9676166591 (127.9.231.65). New cstate: current_term: 1 leader_uuid: "db72f0aea2464a0e98600f9676166591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db72f0aea2464a0e98600f9676166591" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 } health_report { overall_health: UNKNOWN } } }
I20260620 02:25:31.948714 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:31.942754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.9.231.65:42577)
0620 02:25:31.943036 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:31.943042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:31.943226 (+ 184us) server_negotiation.cc:407] Connection header received
0620 02:25:31.943501 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:31.943505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:31.943600 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:31.943760 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:25:31.944174 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.944923 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.946403 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:31.946838 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:31.946983 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:31.947565 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:25:31.948108 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:25:31.948276 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:31.948336 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 02:25:31.948425 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 02:25:31.957865 26700 raft_consensus.cc:1261] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Refusing update from remote peer db72f0aea2464a0e98600f9676166591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:25:31.957870 26569 raft_consensus.cc:1261] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 1 FOLLOWER]: Refusing update from remote peer db72f0aea2464a0e98600f9676166591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:25:31.959008 26768 consensus_queue.cc:1035] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d693bcee1b1d4dd18d25717c1d3e694f" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:25:31.959412 26772 consensus_queue.cc:1035] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4803b65224c4ab7a3f537507a974404" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 34775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:25:31.993166 26297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:25:31.993523 26297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:25:31.997169 26297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Sending DeleteTablet for 3 replicas of tablet 79b259cb872f4b5eae54c77054d4f3fb
I20260620 02:25:31.998590 26549 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b259cb872f4b5eae54c77054d4f3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:25:31 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:48854
I20260620 02:25:31.998590 26680 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b259cb872f4b5eae54c77054d4f3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:25:31 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:46392
I20260620 02:25:31.999281 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 26334
I20260620 02:25:31.999430 26417 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b259cb872f4b5eae54c77054d4f3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:25:31 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:37022
I20260620 02:25:32.013728 26786 tablet_replica.cc:330] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: stopping tablet replica
I20260620 02:25:32.014894 26787 tablet_replica.cc:330] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: stopping tablet replica
I20260620 02:25:32.015379 26787 raft_consensus.cc:2229] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:25:32.015956 26787 raft_consensus.cc:2258] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:25:32.015987 26788 tablet_replica.cc:330] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: stopping tablet replica
I20260620 02:25:32.014272 26786 raft_consensus.cc:2229] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:25:32.016410 26788 raft_consensus.cc:2229] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:25:32.016839 26786 raft_consensus.cc:2258] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:25:32.017097 26788 raft_consensus.cc:2258] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:25:32.018859 26787 ts_tablet_manager.cc:1905] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:25:32.020164 26786 ts_tablet_manager.cc:1905] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:25:32.025475 26788 ts_tablet_manager.cc:1905] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:25:32.027486 26786 ts_tablet_manager.cc:1918] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:25:32.027673 26786 log.cc:1196] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/79b259cb872f4b5eae54c77054d4f3fb
I20260620 02:25:32.028246 26786 ts_tablet_manager.cc:1939] T 79b259cb872f4b5eae54c77054d4f3fb P f4803b65224c4ab7a3f537507a974404: Deleting consensus metadata
I20260620 02:25:32.030333 26281 catalog_manager.cc:4841] TS f4803b65224c4ab7a3f537507a974404 (127.9.231.66:34775): tablet 79b259cb872f4b5eae54c77054d4f3fb (table test-table [id=3b76c6aaaa524c8aa468a7d9a21615cb]) successfully deleted
I20260620 02:25:32.034124 26788 ts_tablet_manager.cc:1918] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:25:32.034366 26788 log.cc:1196] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/79b259cb872f4b5eae54c77054d4f3fb
I20260620 02:25:32.036201 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:32.000513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.9.231.65:42577)
0620 02:25:32.000761 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:32.000770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:32.010091 (+ 9321us) server_negotiation.cc:407] Connection header received
0620 02:25:32.010608 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:32.010614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:32.010701 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:32.010777 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:32.017378 (+ 6601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:32.018647 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:32.021340 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:32.022043 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:32.022164 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:32.026548 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:32.026564 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:32.026581 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:32.026733 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:32.032194 (+ 5461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:32.033350 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:32.033361 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:32.033366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:32.033571 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:32.034126 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:32.034132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:32.034135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:32.034689 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:32.035115 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:32.035494 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 02:25:32.035769 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:25:32.036937 26477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:32.035027 26788 ts_tablet_manager.cc:1939] T 79b259cb872f4b5eae54c77054d4f3fb P db72f0aea2464a0e98600f9676166591: Deleting consensus metadata
I20260620 02:25:32.039894 26787 ts_tablet_manager.cc:1918] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:25:32.040185 26787 log.cc:1196] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/79b259cb872f4b5eae54c77054d4f3fb
I20260620 02:25:32.041147 26787 ts_tablet_manager.cc:1939] T 79b259cb872f4b5eae54c77054d4f3fb P d693bcee1b1d4dd18d25717c1d3e694f: Deleting consensus metadata
I20260620 02:25:32.043023 26282 catalog_manager.cc:4841] TS d693bcee1b1d4dd18d25717c1d3e694f (127.9.231.67:34801): tablet 79b259cb872f4b5eae54c77054d4f3fb (table test-table [id=3b76c6aaaa524c8aa468a7d9a21615cb]) successfully deleted
I20260620 02:25:32.140159 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 26334
I20260620 02:25:32.140463 26283 catalog_manager.cc:4841] TS db72f0aea2464a0e98600f9676166591 (127.9.231.65:42577): tablet 79b259cb872f4b5eae54c77054d4f3fb (table test-table [id=3b76c6aaaa524c8aa468a7d9a21615cb]) successfully deleted
I20260620 02:25:32.160122 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 26486
I20260620 02:25:32.174654 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:32.160973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.9.231.66:34775)
0620 02:25:32.161160 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:32.161164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:32.161183 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:25:32.161301 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:32.161305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:32.161379 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:32.161468 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:32.162930 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:32.163838 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:32.165115 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:32.165533 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:32.165640 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:32.168382 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:32.168399 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:32.168414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:32.168453 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:32.171592 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:32.172334 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:32.172341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:32.172343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:32.172479 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:32.173068 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:32.173071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:32.173073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:32.173469 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:32.173767 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:32.174163 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 02:25:32.174329 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:25:32.175427 26609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:32.268867 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 26486
I20260620 02:25:32.288335 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 26617
I20260620 02:25:32.304697 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:32.289264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.9.231.67:34801)
0620 02:25:32.289469 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:25:32.289474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:25:32.289496 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:25:32.289862 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:25:32.289866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:25:32.289940 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:25:32.290065 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:25:32.291809 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:32.292570 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:32.294014 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:25:32.294584 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:25:32.294711 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:25:32.298030 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:25:32.298048 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:25:32.298065 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:25:32.298121 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:25:32.301257 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:32.302069 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:32.302077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:32.302079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:32.302255 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:25:32.302735 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:25:32.302739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:25:32.302741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:25:32.303321 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:25:32.303566 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:25:32.303996 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 02:25:32.304208 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:25:32.305488 26740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:32.398000 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 26617
I20260620 02:25:32.416383 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 26264
I20260620 02:25:32.417706 26327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:25:32.537447 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 26264
I20260620 02:25:52.562198 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33803
--webserver_interface=127.9.231.126
--webserver_port=41275
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:25:52.930583 26794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:52.930994 26794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:52.931221 26794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:52.939254 26794 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:25:52.939404 26794 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:25:52.939476 26794 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:25:52.939553 26794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:25:52.939636 26794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:52.939742 26794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:52.939832 26794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:25:52.939915 26794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:25:52.950743 26794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=41275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:52.952257 26794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:52.953732 26794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:52.967658 26800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:52.967650 26802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:52.967650 26794 server_base.cc:1034] running on GCE node
W20260620 02:25:52.968102 26799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:52.969066 26794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:52.970679 26794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:52.971940 26794 hybrid_clock.cc:648] HybridClock initialized: now 1781922352971876 us; error 44 us; skew 500 ppm
I20260620 02:25:52.972299 26794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:52 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922352, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:52.982235 26794 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:25:52.984174 26794 webserver.cc:460] Webserver started at http://127.9.231.126:41275/ using document root <none> and password file <none>
I20260620 02:25:52.984969 26794 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:52.985100 26794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:52.991258 26794 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:25:52.994364 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:52.995790 26794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:25:52.995994 26794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "0da91db920e446e78249e22aed2fcd3c"
format_stamp: "Formatted at 2026-06-20 02:25:06 on dist-test-slave-9vg1"
server_key: "573a0e83790c3cb6a5036169dc738d38"
server_key_iv: "1e727a82e89488f7ef8ed2884777713b"
server_key_version: "encryptionkey@0"
I20260620 02:25:52.997002 26794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:53.012409 26794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:53.016856 26794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:53.017189 26794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:53.038096 26794 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33803
I20260620 02:25:53.038115 26860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33803 every 8 connection(s)
I20260620 02:25:53.039846 26794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:25:53.047876 26861 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:25:53.050038 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 26794
I20260620 02:25:53.051334 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:42577
--local_ip_for_outbound_sockets=127.9.231.65
--tserver_master_addrs=127.9.231.126:33803
--webserver_port=33551
--webserver_interface=127.9.231.65
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:25:53.053225 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Bootstrap starting.
I20260620 02:25:53.079037 26861 log.cc:824] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:25:53.097978 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:25:53.098683 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Bootstrap complete.
I20260620 02:25:53.110028 26861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:53.111482 26861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da91db920e446e78249e22aed2fcd3c, State: Initialized, Role: FOLLOWER
I20260620 02:25:53.112200 26861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:53.112474 26861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:25:53.112792 26861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:25:53.113065 26861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:25:53.117274 26861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:53.117870 26861 leader_election.cc:302] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da91db920e446e78249e22aed2fcd3c; no voters:
I20260620 02:25:53.118643 26861 leader_election.cc:288] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:25:53.118793 26865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:25:53.120172 26865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [term 3 LEADER]: Becoming Leader. State: Replica: 0da91db920e446e78249e22aed2fcd3c, State: Running, Role: LEADER
I20260620 02:25:53.120834 26865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } }
I20260620 02:25:53.121335 26861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:25:53.123737 26867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da91db920e446e78249e22aed2fcd3c. Latest consensus state: current_term: 3 leader_uuid: "0da91db920e446e78249e22aed2fcd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } } }
I20260620 02:25:53.124783 26867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:25:53.124257 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0da91db920e446e78249e22aed2fcd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da91db920e446e78249e22aed2fcd3c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33803 } } }
I20260620 02:25:53.124965 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:25:53.137319 26871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:25:53.147733 26871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:25:53.149443 26871 catalog_manager.cc:1259] Loaded cluster ID: f82897b736de45278969cfcfe2cbc7d8
I20260620 02:25:53.149641 26871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:25:53.155103 26871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:25:53.156948 26871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Loaded TSK: 0
I20260620 02:25:53.157234 26871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0da91db920e446e78249e22aed2fcd3c: Loaded TSK: 1
F20260620 02:25:53.158538 26871 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa512da99ad google::LogMessage::Fail() at ??:0
@ 0x7fa512dac88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa512da94de google::LogMessage::Flush() at ??:0
@ 0x7fa512daaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa516b0d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa516afc73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa5287dc1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa5287db783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa5288672a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa5288670c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa5288b15e9 std::function<>::operator()() at ??:0
@ 0x7fa5287dfcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa5287ddc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa528865610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa528865439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa5159f5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa5159f891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa5159f8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa5159d3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa518cad6db start_thread at ??:0
@ 0x7fa511c4361f clone at ??:0
W20260620 02:25:53.481737 26863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:53.482103 26863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:53.482281 26863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:53.490113 26863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:53.490312 26863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:53.490523 26863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:25:53.501555 26863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:42577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=33551
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:53.503135 26863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:53.504801 26863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:53.520148 26908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:53.520313 26909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:53.520337 26911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:53.521636 26863 server_base.cc:1034] running on GCE node
I20260620 02:25:53.522436 26863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:53.523590 26863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:53.524839 26863 hybrid_clock.cc:648] HybridClock initialized: now 1781922353524712 us; error 122 us; skew 500 ppm
I20260620 02:25:53.525238 26863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:53 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922353, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:53.529002 26863 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:25:53.530685 26863 webserver.cc:460] Webserver started at http://127.9.231.65:33551/ using document root <none> and password file <none>
I20260620 02:25:53.531575 26863 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:53.531755 26863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:53.537967 26863 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:25:53.541363 26918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:53.543306 26863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:25:53.543524 26863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "db72f0aea2464a0e98600f9676166591"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "65de3fd291e60a80ce4ba01e814e9600"
server_key_iv: "04c840f5cb9a2e4487f694974671d06b"
server_key_version: "encryptionkey@0"
I20260620 02:25:53.544667 26863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:53.558961 26863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:53.564075 26863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:53.564410 26863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:53.565630 26863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:53.567381 26863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:53.567530 26863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:53.567685 26863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:53.567790 26863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:53.604676 26863 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:42577
I20260620 02:25:53.604802 27030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:42577 every 8 connection(s)
I20260620 02:25:53.606357 26863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:25:53.612114 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:53.609946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:54083)
0620 02:25:53.610826 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:53.611166 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":331,"threads_started":1}
W20260620 02:25:53.612933 27031 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:33803 (0 consecutive failures): Network error: Failed to ping master at 127.9.231.126:33803: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
I20260620 02:25:53.613958 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:53.613542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:39463)
0620 02:25:53.613749 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:53.613832 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:25:53.614347 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 26863
I20260620 02:25:53.615024 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:34775
--local_ip_for_outbound_sockets=127.9.231.66
--tserver_master_addrs=127.9.231.126:33803
--webserver_port=41761
--webserver_interface=127.9.231.66
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:25:53.615302 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:53.614774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:42495)
0620 02:25:53.614947 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:53.615045 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 02:25:53.615571 27031 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:25:53.969986 27034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:53.970355 27034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:53.970525 27034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:53.978412 27034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:53.978603 27034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:53.978791 27034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:25:53.989347 27034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:34775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=41761
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:53.990795 27034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:53.992257 27034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:54.007220 27039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:54.007231 27042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:54.007227 27040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:54.007972 27034 server_base.cc:1034] running on GCE node
I20260620 02:25:54.008668 27034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:54.009732 27034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:54.010977 27034 hybrid_clock.cc:648] HybridClock initialized: now 1781922354010923 us; error 43 us; skew 500 ppm
I20260620 02:25:54.011390 27034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:54 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:54.014901 27034 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:25:54.016592 27034 webserver.cc:460] Webserver started at http://127.9.231.66:41761/ using document root <none> and password file <none>
I20260620 02:25:54.017357 27034 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:54.017519 27034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:54.023792 27034 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:25:54.027019 27049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:54.028733 27034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:25:54.028975 27034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "f4803b65224c4ab7a3f537507a974404"
format_stamp: "Formatted at 2026-06-20 02:25:07 on dist-test-slave-9vg1"
server_key: "a6fa261f1ca79eb3aefe2c10db0086a4"
server_key_iv: "7317ab94c946b15dfc97952b3262183f"
server_key_version: "encryptionkey@0"
I20260620 02:25:54.030097 27034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:54.044787 27034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:54.050235 27034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:54.050495 27034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:54.051676 27034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:54.053282 27034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:54.053428 27034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:54.053577 27034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:54.053674 27034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:54.088235 27034 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:34775
I20260620 02:25:54.088335 27161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:34775 every 8 connection(s)
I20260620 02:25:54.089830 27034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:25:54.092929 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27034
I20260620 02:25:54.093791 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:34801
--local_ip_for_outbound_sockets=127.9.231.67
--tserver_master_addrs=127.9.231.126:33803
--webserver_port=46245
--webserver_interface=127.9.231.67
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:25:54.096868 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:54.094003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:47191)
0620 02:25:54.095004 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:54.095365 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":388,"threads_started":1}
W20260620 02:25:54.097697 27162 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:33803 (0 consecutive failures): Network error: Failed to ping master at 127.9.231.126:33803: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
I20260620 02:25:54.098851 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:54.098437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:37009)
0620 02:25:54.098639 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:54.098722 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:25:54.099923 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:54.099530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:52497)
0620 02:25:54.099696 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:54.099788 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 02:25:54.100194 27162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:25:54.455698 27165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:25:54.456025 27165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:25:54.456197 27165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:25:54.464061 27165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:25:54.464198 27165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:25:54.464383 27165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:25:54.474864 27165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:34801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=46245
--tserver_master_addrs=127.9.231.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:25:54.476378 27165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:25:54.477818 27165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:25:54.492274 27173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:54.492316 27170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:25:54.492341 27171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:25:54.493878 27165 server_base.cc:1034] running on GCE node
I20260620 02:25:54.494604 27165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:25:54.495723 27165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:25:54.496940 27165 hybrid_clock.cc:648] HybridClock initialized: now 1781922354496888 us; error 44 us; skew 500 ppm
I20260620 02:25:54.497282 27165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:25:54 dist-test-slave-9vg1 krb5kdc[25711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:25:54.500928 27165 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:25:54.502784 27165 webserver.cc:460] Webserver started at http://127.9.231.67:46245/ using document root <none> and password file <none>
I20260620 02:25:54.503643 27165 fs_manager.cc:359] Metadata directory not provided
I20260620 02:25:54.503814 27165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:25:54.510030 27165 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:25:54.513307 27180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:54.514865 27165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:25:54.515136 27165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "d693bcee1b1d4dd18d25717c1d3e694f"
format_stamp: "Formatted at 2026-06-20 02:25:08 on dist-test-slave-9vg1"
server_key: "e61d21b4aa6bc878055aa9bac073eeac"
server_key_iv: "dd916a2c7642bbb7f8d009336715c783"
server_key_version: "encryptionkey@0"
I20260620 02:25:54.516474 27165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:25:54.545390 27165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:25:54.551226 27165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:25:54.551610 27165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:25:54.553246 27165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:25:54.555455 27165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:25:54.555583 27165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:54.555711 27165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:25:54.555780 27165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:25:54.600032 27165 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:34801
I20260620 02:25:54.600052 27292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:34801 every 8 connection(s)
I20260620 02:25:54.602646 27165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:25:54.604337 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27165
I20260620 02:25:54.612581 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:54.610473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:41015)
0620 02:25:54.611445 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:54.611697 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":400,"threads_started":1}
W20260620 02:25:54.613567 27293 heartbeater.cc:644] Failed to heartbeat to 127.9.231.126:33803 (0 consecutive failures): Network error: Failed to ping master at 127.9.231.126:33803: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
I20260620 02:25:54.614575 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:54.614250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:52947)
0620 02:25:54.614390 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:54.614463 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 02:25:54.615607 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:54.615256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:32809)
0620 02:25:54.615408 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:54.615492 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:25:54.615890 27293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:25:54.617055 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:54.616133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:52083)
0620 02:25:54.616748 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:54.616831 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":275,"threads_started":1}
I20260620 02:25:55.101936 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:55.100979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:35575)
0620 02:25:55.101516 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:55.101668 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":253,"threads_started":1}
I20260620 02:25:55.617638 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:55.616606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:51153)
0620 02:25:55.617167 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:55.617274 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":264,"threads_started":1}
I20260620 02:25:55.618991 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:55.618122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:45421)
0620 02:25:55.618723 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:55.618819 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":241,"threads_started":1}
I20260620 02:25:56.104187 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:56.103292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:37403)
0620 02:25:56.103853 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:56.103973 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":314,"threads_started":1}
I20260620 02:25:56.619905 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:56.618894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:47933)
0620 02:25:56.619524 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:56.619633 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":357,"threads_started":1}
I20260620 02:25:56.620782 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:56.620033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:38337)
0620 02:25:56.620523 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:56.620603 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":275,"threads_started":1}
I20260620 02:25:57.106604 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:57.105637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:57499)
0620 02:25:57.106293 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:57.106385 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":337,"threads_started":1}
I20260620 02:25:57.621999 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:57.620958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:48733)
0620 02:25:57.621625 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:57.621748 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":378,"threads_started":1}
I20260620 02:25:57.622758 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:57.621855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:60933)
0620 02:25:57.622416 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:57.622522 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":303,"threads_started":1}
I20260620 02:25:58.108491 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:58.107701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:44613)
0620 02:25:58.108202 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:58.108286 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
I20260620 02:25:58.624197 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:58.623360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:35743)
0620 02:25:58.623917 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:58.624006 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":313,"threads_started":1}
I20260620 02:25:58.624850 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:58.623841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:60563)
0620 02:25:58.624511 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:58.624670 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":311,"threads_started":1}
I20260620 02:25:59.110584 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:59.109720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:39229)
0620 02:25:59.110273 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:59.110358 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":286,"threads_started":1}
I20260620 02:25:59.626210 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:59.625338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:34267)
0620 02:25:59.625933 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:59.626028 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":362,"threads_started":1}
I20260620 02:25:59.626760 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:25:59.625853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:51019)
0620 02:25:59.626402 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 02:25:59.626523 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":289,"threads_started":1}
I20260620 02:26:00.112596 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:00.111664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:52385)
0620 02:26:00.112205 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:00.112317 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":275,"threads_started":1}
I20260620 02:26:00.628372 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:00.627403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:55091)
0620 02:26:00.628009 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:00.628162 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":283,"threads_started":1}
I20260620 02:26:00.628723 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:00.627780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:44421)
0620 02:26:00.628301 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:00.628383 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":218,"threads_started":1}
I20260620 02:26:01.114625 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:01.113743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:50105)
0620 02:26:01.114296 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:01.114392 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":259,"threads_started":1}
I20260620 02:26:01.630625 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:01.629579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:38617)
0620 02:26:01.630212 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:01.630348 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":274,"threads_started":1}
I20260620 02:26:01.630877 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:01.630020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:54279)
0620 02:26:01.630495 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:01.630648 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":273,"threads_started":1}
I20260620 02:26:02.116583 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:02.115790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:53945)
0620 02:26:02.116310 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:02.116399 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":267,"threads_started":1}
I20260620 02:26:02.632985 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:02.631908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:56325)
0620 02:26:02.632513 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:02.632716 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
I20260620 02:26:02.633359 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:02.631869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:57497)
0620 02:26:02.632874 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:02.632957 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":753,"threads_started":1}
I20260620 02:26:03.118891 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:03.117855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:35979)
0620 02:26:03.118607 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:03.118703 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":293,"threads_started":1}
I20260620 02:26:03.635214 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:03.634241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:39103)
0620 02:26:03.634836 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:03.634927 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":347,"threads_started":1}
I20260620 02:26:03.635439 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:03.634383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:41813)
0620 02:26:03.635118 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:03.635220 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":371,"threads_started":1}
I20260620 02:26:04.121230 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:04.120211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:33685)
0620 02:26:04.120870 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:04.120964 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":338,"threads_started":1}
I20260620 02:26:04.637356 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:04.636391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:59491)
0620 02:26:04.636949 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:04.637060 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":271,"threads_started":1}
I20260620 02:26:04.639822 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:04.636343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:40537)
0620 02:26:04.639462 (+ 3119us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:04.639551 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3012,"thread_start_us":2923,"threads_started":1}
I20260620 02:26:05.123301 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:05.122311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:47035)
0620 02:26:05.122840 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:05.122939 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260620 02:26:05.639570 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:05.638624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:58257)
0620 02:26:05.639297 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:05.639393 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":299,"threads_started":1}
I20260620 02:26:05.641835 27330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:05.640842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:33925)
0620 02:26:05.641359 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:05.641496 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":279,"threads_started":1}
I20260620 02:26:06.125389 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:06.124484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:42993)
0620 02:26:06.125112 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:06.125213 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":344,"threads_started":1}
I20260620 02:26:06.641712 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:06.640844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:45569)
0620 02:26:06.641384 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:06.641528 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":284,"threads_started":1}
I20260620 02:26:06.643769 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:06.642858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:37965)
0620 02:26:06.643433 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:06.643537 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":350,"threads_started":1}
I20260620 02:26:07.128032 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:07.127116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:59705)
0620 02:26:07.127721 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:07.127834 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":277,"threads_started":1}
I20260620 02:26:07.643824 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:07.642921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:51887)
0620 02:26:07.643526 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:07.643626 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":317,"threads_started":1}
I20260620 02:26:07.645550 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:07.644718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:52109)
0620 02:26:07.645257 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:07.645355 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":303,"threads_started":1}
I20260620 02:26:08.130190 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:08.129317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:41629)
0620 02:26:08.129898 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:08.129990 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":320,"threads_started":1}
I20260620 02:26:08.645989 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:08.645175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:48639)
0620 02:26:08.645705 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:08.645808 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":241,"threads_started":1}
I20260620 02:26:08.647293 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:08.646510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:59277)
0620 02:26:08.646928 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:08.647024 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":209,"threads_started":1}
I20260620 02:26:09.132618 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:09.131751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.66:60871)
0620 02:26:09.132307 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:09.132413 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":318,"threads_started":1}
I20260620 02:26:09.606614 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 26863
I20260620 02:26:09.634246 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:09.607949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.9.231.65:42577)
0620 02:26:09.609183 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:09.609206 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:09.609260 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:26:09.610995 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:09.611007 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:09.611550 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:09.611876 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:09.613681 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:09.615038 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:09.616633 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:09.617425 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:09.617590 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:09.620871 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:09.620881 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:09.620901 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:09.620951 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:09.629855 (+ 8904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:09.630892 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:09.630902 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:09.630912 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:09.631249 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:09.631765 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:09.631769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:09.631771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:09.632412 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:09.632706 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:09.633111 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 02:26:09.633740 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":850,"threads_started":1}
I20260620 02:26:09.636106 27026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:09.649541 27343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:09.648627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.67:50787)
0620 02:26:09.649236 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:09.649355 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":301,"threads_started":1}
I20260620 02:26:09.726212 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:09.724896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33803 (local address 127.9.231.65:52821)
0620 02:26:09.725556 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:09.725825 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.126:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":341,"threads_started":1}
I20260620 02:26:09.726241 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 26863
I20260620 02:26:09.745296 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27034
I20260620 02:26:09.765908 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:09.746575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.9.231.66:34775)
0620 02:26:09.747241 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:09.747256 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:09.747296 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:26:09.748910 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:09.748920 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:09.749368 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:09.749684 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:09.751562 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:09.753226 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:09.754816 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:09.755669 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:09.755907 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:09.758727 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:09.758735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:09.758751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:09.758799 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:09.762109 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:09.763033 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:09.763042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:09.763076 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:09.763298 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:09.763735 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:09.763740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:09.763744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:09.764313 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:09.764653 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:09.765145 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 02:26:09.765606 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":292,"threads_started":1}
I20260620 02:26:09.767405 27157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:09.859187 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27034
I20260620 02:26:09.878659 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27165
I20260620 02:26:09.899312 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:09.879815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.9.231.67:34801)
0620 02:26:09.880416 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:09.880432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:09.880475 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:26:09.882162 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:09.882174 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:09.882625 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:09.883022 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:09.884625 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:09.885895 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:09.887383 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:09.888107 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:09.888334 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:09.891552 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:09.891562 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:09.891581 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:09.891631 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:09.894830 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:09.895813 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:09.895822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:09.895829 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:09.896037 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:09.896496 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:09.896501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:09.896516 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:09.897133 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:09.897502 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:09.897994 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 02:26:09.898793 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":279,"threads_started":1}
I20260620 02:26:09.900887 27288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:09.989714 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27165
2026-06-20T02:26:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65298 ms)
[----------] 30 tests from SecurityITest (314665 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:10 dist-test-slave-9vg1 krb5kdc[27351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:10 dist-test-slave-9vg1 krb5kdc[27351](info): set up 2 sockets
Jun 20 02:26:10 dist-test-slave-9vg1 krb5kdc[27351](info): commencing operation
krb5kdc: starting...
W20260620 02:26:11.144402 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.009s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:11 dist-test-slave-9vg1 krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:26:11.425455 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38035
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:11.784385 27364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:11.784723 27364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:11.784894 27364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:11.793370 27364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:11.793511 27364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:11.793601 27364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:11.793709 27364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:11.793805 27364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:11.804745 27364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:11.806241 27364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:11.807907 27364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:11.821671 27369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:11.821677 27370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:11.821648 27372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:11.822436 27364 server_base.cc:1034] running on GCE node
I20260620 02:26:11.823602 27364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:11.825337 27364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:11.826618 27364 hybrid_clock.cc:648] HybridClock initialized: now 1781922371826568 us; error 41 us; skew 500 ppm
I20260620 02:26:11.827018 27364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:11 dist-test-slave-9vg1 krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922371, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:11.830730 27364 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:26:11.832662 27364 webserver.cc:460] Webserver started at http://127.9.231.126:38113/ using document root <none> and password file <none>
I20260620 02:26:11.833474 27364 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:11.833648 27364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:11.834002 27364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:11.837482 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "2d327ac56eac4b2fadc65aa25c15445c"
format_stamp: "Formatted at 2026-06-20 02:26:11 on dist-test-slave-9vg1"
server_key: "c621e33eb9d47e91d05ae6ba96863bde"
server_key_iv: "628dc4db9c5422aaa0f2a2a54cf87650"
server_key_version: "encryptionkey@0"
I20260620 02:26:11.838392 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "2d327ac56eac4b2fadc65aa25c15445c"
format_stamp: "Formatted at 2026-06-20 02:26:11 on dist-test-slave-9vg1"
server_key: "c621e33eb9d47e91d05ae6ba96863bde"
server_key_iv: "628dc4db9c5422aaa0f2a2a54cf87650"
server_key_version: "encryptionkey@0"
I20260620 02:26:11.844569 27364 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260620 02:26:11.848771 27379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:11.850432 27364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:26:11.850690 27364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "2d327ac56eac4b2fadc65aa25c15445c"
format_stamp: "Formatted at 2026-06-20 02:26:11 on dist-test-slave-9vg1"
server_key: "c621e33eb9d47e91d05ae6ba96863bde"
server_key_iv: "628dc4db9c5422aaa0f2a2a54cf87650"
server_key_version: "encryptionkey@0"
I20260620 02:26:11.850958 27364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:11.867971 27364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:11.873489 27364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:11.873900 27364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:11.895926 27364 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38035
I20260620 02:26:11.895949 27430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38035 every 8 connection(s)
I20260620 02:26:11.897684 27364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:11.901999 27431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:11.904191 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27364
I20260620 02:26:11.904431 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:11.905289 10141 external_mini_cluster.cc:1428] Setting key ec0bc91493fe54bbfa70cc90bcac11f4
I20260620 02:26:11.915984 27431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c: Bootstrap starting.
I20260620 02:26:11.920404 27431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:11.921577 27431 log.cc:824] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:26:11 dist-test-slave-9vg1 krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:11.925498 27431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c: No bootstrap required, opened a new log
I20260620 02:26:11.933894 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:11.908008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.9.231.126:38035)
0620 02:26:11.909048 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:11.909062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:11.909111 (+ 49us) server_negotiation.cc:407] Connection header received
0620 02:26:11.911353 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:11.911381 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:11.911883 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:11.912302 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:11.914320 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:11.916014 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:11.918207 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:11.919047 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:11.919343 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:11.924431 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:11.924517 (+ 86us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:11.924535 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:11.924591 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:11.928652 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:11.929783 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:11.929791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:11.929799 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:11.930017 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:11.930483 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:11.930486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:11.930488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:11.931299 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:11.931536 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:11.932124 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 02:26:11.932595 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":501,"threads_started":1}
I20260620 02:26:11.935482 27431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d327ac56eac4b2fadc65aa25c15445c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38035 } }
I20260620 02:26:11.935915 27431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:11.936045 27431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d327ac56eac4b2fadc65aa25c15445c, State: Initialized, Role: FOLLOWER
I20260620 02:26:11.936671 27431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d327ac56eac4b2fadc65aa25c15445c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38035 } }
I20260620 02:26:11.936996 27431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:11.937181 27431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:11.937371 27431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:11.940227 27431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d327ac56eac4b2fadc65aa25c15445c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38035 } }
I20260620 02:26:11.940732 27431 leader_election.cc:302] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d327ac56eac4b2fadc65aa25c15445c; no voters:
I20260620 02:26:11.941308 27431 leader_election.cc:288] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:11.941475 27436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:11.942283 27436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [term 1 LEADER]: Becoming Leader. State: Replica: 2d327ac56eac4b2fadc65aa25c15445c, State: Running, Role: LEADER
I20260620 02:26:11.942888 27436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d327ac56eac4b2fadc65aa25c15445c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38035 } }
I20260620 02:26:11.943698 27431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:11.945607 27437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d327ac56eac4b2fadc65aa25c15445c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d327ac56eac4b2fadc65aa25c15445c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38035 } } }
I20260620 02:26:11.946069 27437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:11.957510 27438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d327ac56eac4b2fadc65aa25c15445c. Latest consensus state: current_term: 1 leader_uuid: "2d327ac56eac4b2fadc65aa25c15445c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d327ac56eac4b2fadc65aa25c15445c" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38035 } } }
I20260620 02:26:11.958145 27438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:11.959232 27443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:11.964587 27443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:11.974097 27443 catalog_manager.cc:1347] Generated new cluster ID: cc6dec1833cd46988f1644c127c4de3e
I20260620 02:26:11.974258 27443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:11.988549 27443 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:11.990690 27443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:12.004528 27443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c: Generated new TSK 0
I20260620 02:26:12.005663 27443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:26:12.079619 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38035
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:12.447391 27458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:12.447714 27458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:12.447922 27458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:12.455928 27458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:12.456168 27458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:12.456434 27458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:26:12.467656 27458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:12.468973 27458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:12.470234 27458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:12.486265 27466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:12.486210 27464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:12.486240 27463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:12.487067 27458 server_base.cc:1034] running on GCE node
I20260620 02:26:12.487703 27458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:12.488708 27458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:12.489954 27458 hybrid_clock.cc:648] HybridClock initialized: now 1781922372489883 us; error 56 us; skew 500 ppm
I20260620 02:26:12.490367 27458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:12 dist-test-slave-9vg1 krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:12.494151 27458 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:26:12.495996 27458 webserver.cc:460] Webserver started at http://127.9.231.65:37743/ using document root <none> and password file <none>
I20260620 02:26:12.496876 27458 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:12.497037 27458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:12.497383 27458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:12.500653 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "b6ea89528d294d51975c8552e85f4f97"
format_stamp: "Formatted at 2026-06-20 02:26:12 on dist-test-slave-9vg1"
server_key: "f3b7f2850fd20edce53288c953c4b8b5"
server_key_iv: "e6d02fc7681822b8c782f0d03f3403d8"
server_key_version: "encryptionkey@0"
I20260620 02:26:12.501462 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "b6ea89528d294d51975c8552e85f4f97"
format_stamp: "Formatted at 2026-06-20 02:26:12 on dist-test-slave-9vg1"
server_key: "f3b7f2850fd20edce53288c953c4b8b5"
server_key_iv: "e6d02fc7681822b8c782f0d03f3403d8"
server_key_version: "encryptionkey@0"
I20260620 02:26:12.507197 27458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:26:12.511171 27473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:12.512845 27458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:26:12.513060 27458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "b6ea89528d294d51975c8552e85f4f97"
format_stamp: "Formatted at 2026-06-20 02:26:12 on dist-test-slave-9vg1"
server_key: "f3b7f2850fd20edce53288c953c4b8b5"
server_key_iv: "e6d02fc7681822b8c782f0d03f3403d8"
server_key_version: "encryptionkey@0"
I20260620 02:26:12.513315 27458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:12.539628 27458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:12.544365 27458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:12.544701 27458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:12.545749 27458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:12.547430 27458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:12.547545 27458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:12.547708 27458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:12.547784 27458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:12.584044 27458 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:42597
I20260620 02:26:12.584110 27585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:42597 every 8 connection(s)
I20260620 02:26:12.585426 27458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:26:12.588604 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27458
I20260620 02:26:12.588845 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:26:12.589730 10141 external_mini_cluster.cc:1428] Setting key d99dd8af25f824f6cf18a2e379ee929f
Jun 20 02:26:12 dist-test-slave-9vg1 krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:12.610042 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:12.588277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:57451 (local address 127.9.231.126:38035)
0620 02:26:12.588807 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:12.588813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:12.590177 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 02:26:12.592844 (+ 2667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:12.592850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:12.592999 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:12.593111 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:12.595123 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:12.595897 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:12.597671 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:12.598107 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:12.598275 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:12.602757 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:12.602763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:12.602766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:12.602804 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:12.605853 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:12.607025 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:12.607030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:12.607033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:12.607302 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:12.607918 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:12.607923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:12.607925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:12.608266 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:12.608389 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:12.609554 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0620 02:26:12.609765 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":264,"threads_started":1}
I20260620 02:26:12.611805 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:12.589080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38035 (local address 127.9.231.65:57451)
0620 02:26:12.589997 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:12.590038 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:12.592488 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:12.593629 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:12.593641 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:12.594251 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:12.594887 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:12.594906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:12.596094 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:12.596100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:12.597475 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:12.597489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:12.598385 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:12.598394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:12.598672 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:12.599507 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:12.599553 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:12.602436 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:12.606043 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:12.606056 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:12.606082 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:12.606822 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:12.607435 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:12.607443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:12.607449 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:12.607754 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:12.608470 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:12.608483 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:12.609006 (+ 523us) client_negotiation.cc:769] Sending connection context
0620 02:26:12.609430 (+ 424us) client_negotiation.cc:240] Negotiation successful
0620 02:26:12.610078 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":336,"threads_started":1}
I20260620 02:26:12.613680 27586 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38035
I20260620 02:26:12.614085 27586 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:12.615033 27586 heartbeater.cc:511] Master 127.9.231.126:38035 requested a full tablet report, sending...
I20260620 02:26:12.618032 27396 ts_manager.cc:194] Registered new tserver with Master: b6ea89528d294d51975c8552e85f4f97 (127.9.231.65:42597)
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:26:12.619947 27396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:57451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:26:12.655665 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38035
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:13.024595 27592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:13.024905 27592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:13.025012 27592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:13.032832 27592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:13.033026 27592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:13.033263 27592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:26:13.045317 27592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:13.046743 27592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:13.048463 27592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:13.064369 27598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:13.064424 27600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:13.064774 27597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:13.065713 27592 server_base.cc:1034] running on GCE node
I20260620 02:26:13.066481 27592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:13.067716 27592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:13.069002 27592 hybrid_clock.cc:648] HybridClock initialized: now 1781922373068945 us; error 48 us; skew 500 ppm
I20260620 02:26:13.069399 27592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:13 dist-test-slave-9vg1 krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:13.073225 27592 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:26:13.075026 27592 webserver.cc:460] Webserver started at http://127.9.231.66:34669/ using document root <none> and password file <none>
I20260620 02:26:13.075879 27592 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:13.076037 27592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:13.076380 27592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:13.079664 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "30b266fe402b4845b9222abe7fe253ff"
format_stamp: "Formatted at 2026-06-20 02:26:13 on dist-test-slave-9vg1"
server_key: "91b761e76efcf60d406f04b25f0595c8"
server_key_iv: "4663f5937a074ef8e81429f37c2f906e"
server_key_version: "encryptionkey@0"
I20260620 02:26:13.080495 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "30b266fe402b4845b9222abe7fe253ff"
format_stamp: "Formatted at 2026-06-20 02:26:13 on dist-test-slave-9vg1"
server_key: "91b761e76efcf60d406f04b25f0595c8"
server_key_iv: "4663f5937a074ef8e81429f37c2f906e"
server_key_version: "encryptionkey@0"
I20260620 02:26:13.086448 27592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:26:13.090428 27607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:13.091976 27592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:13.092181 27592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "30b266fe402b4845b9222abe7fe253ff"
format_stamp: "Formatted at 2026-06-20 02:26:13 on dist-test-slave-9vg1"
server_key: "91b761e76efcf60d406f04b25f0595c8"
server_key_iv: "4663f5937a074ef8e81429f37c2f906e"
server_key_version: "encryptionkey@0"
I20260620 02:26:13.092412 27592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:13.111321 27592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:13.116290 27592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:13.116626 27592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:13.117823 27592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:13.119438 27592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:13.119551 27592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:13.119719 27592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:13.119796 27592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:13.156569 27592 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:35967
I20260620 02:26:13.156621 27719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:35967 every 8 connection(s)
I20260620 02:26:13.158012 27592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:26:13.165282 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27592
I20260620 02:26:13.165480 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:26:13.166000 10141 external_mini_cluster.cc:1428] Setting key bb9d4bcd44d6dc276a452e98752fbfe2
Jun 20 02:26:13 dist-test-slave-9vg1 krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:13.180361 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.160894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:57501 (local address 127.9.231.126:38035)
0620 02:26:13.161323 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.161327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.162328 (+ 1001us) server_negotiation.cc:407] Connection header received
0620 02:26:13.164394 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.164399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.164491 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.164599 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:13.166496 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.167339 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.169028 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.169472 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.169632 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.173465 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:13.173472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:13.173475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:13.173515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:13.176525 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:13.177648 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:13.177693 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:13.177695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:13.177843 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:13.178377 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:13.178380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:13.178382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:13.178703 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:13.178850 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.179869 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.180077 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260620 02:26:13.181998 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.161417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38035 (local address 127.9.231.66:57501)
0620 02:26:13.162144 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:13.162193 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:13.164082 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:13.164856 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:13.164865 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:13.165458 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:13.166237 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.166284 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.167532 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.167536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.168861 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.168873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.169767 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.169777 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.170026 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.170771 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:13.170799 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:13.173170 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:13.176732 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:13.176746 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:13.176771 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:13.177480 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:13.177966 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:13.177974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:13.177980 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:13.178254 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:13.178853 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:13.178865 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:13.179428 (+ 563us) client_negotiation.cc:769] Sending connection context
0620 02:26:13.179788 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 02:26:13.180423 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":292,"threads_started":1}
I20260620 02:26:13.183713 27720 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38035
I20260620 02:26:13.184055 27720 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:13.184862 27720 heartbeater.cc:511] Master 127.9.231.126:38035 requested a full tablet report, sending...
I20260620 02:26:13.186546 27396 ts_manager.cc:194] Registered new tserver with Master: 30b266fe402b4845b9222abe7fe253ff (127.9.231.66:35967)
I20260620 02:26:13.187745 27396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:57501
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:26:13.228258 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38035
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:13.585012 27726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:13.585346 27726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:13.585523 27726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:13.593534 27726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:13.593711 27726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:13.593887 27726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:26:13.604740 27726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:38035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:13.606393 27726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:13.607879 27726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:13.622035 27734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:13.622112 27732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:13.622136 27731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:13.622959 27726 server_base.cc:1034] running on GCE node
I20260620 02:26:13.623736 27726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:13.623986 27586 heartbeater.cc:503] Master 127.9.231.126:38035 was elected leader, sending a full tablet report...
I20260620 02:26:13.624913 27726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:13.626171 27726 hybrid_clock.cc:648] HybridClock initialized: now 1781922373626116 us; error 44 us; skew 500 ppm
I20260620 02:26:13.626528 27726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:13 dist-test-slave-9vg1 krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:13.630121 27726 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:26:13.631966 27726 webserver.cc:460] Webserver started at http://127.9.231.67:41529/ using document root <none> and password file <none>
I20260620 02:26:13.632736 27726 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:13.632915 27726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:13.633261 27726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:13.636482 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "359a6288eb3b488980a27d9385385fe5"
format_stamp: "Formatted at 2026-06-20 02:26:13 on dist-test-slave-9vg1"
server_key: "8f74e35cdf7ce2cdcefbc6a42be0fce1"
server_key_iv: "2829c63aee3abfe0c3b02a3ae5048462"
server_key_version: "encryptionkey@0"
I20260620 02:26:13.637323 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "359a6288eb3b488980a27d9385385fe5"
format_stamp: "Formatted at 2026-06-20 02:26:13 on dist-test-slave-9vg1"
server_key: "8f74e35cdf7ce2cdcefbc6a42be0fce1"
server_key_iv: "2829c63aee3abfe0c3b02a3ae5048462"
server_key_version: "encryptionkey@0"
I20260620 02:26:13.643234 27726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:26:13.647656 27741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:13.649330 27726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:26:13.649546 27726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "359a6288eb3b488980a27d9385385fe5"
format_stamp: "Formatted at 2026-06-20 02:26:13 on dist-test-slave-9vg1"
server_key: "8f74e35cdf7ce2cdcefbc6a42be0fce1"
server_key_iv: "2829c63aee3abfe0c3b02a3ae5048462"
server_key_version: "encryptionkey@0"
I20260620 02:26:13.649788 27726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:13.666436 27726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:13.670991 27726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:13.671376 27726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:13.672411 27726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:13.673974 27726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:13.674086 27726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:13.674243 27726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:13.674324 27726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:13.710292 27726 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:39641
I20260620 02:26:13.710345 27853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:39641 every 8 connection(s)
I20260620 02:26:13.711710 27726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:26:13.715860 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27726
I20260620 02:26:13.716073 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:26:13.716534 10141 external_mini_cluster.cc:1428] Setting key a55ec976f556c8e7e4d1ec8e01cad6cb
Jun 20 02:26:13 dist-test-slave-9vg1 krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:13.734783 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.714428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:37575 (local address 127.9.231.126:38035)
0620 02:26:13.714969 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.714976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.716136 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 02:26:13.718658 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.718663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.718810 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.718929 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:13.720940 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.721764 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.723511 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.723986 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.724091 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.727973 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:13.727979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:13.727983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:13.728022 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:13.731010 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:13.732157 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:13.732162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:13.732165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:13.732361 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:13.732924 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:13.732929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:13.732931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:13.733356 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:13.733484 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.734245 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.734435 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
I20260620 02:26:13.735836 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.714984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38035 (local address 127.9.231.67:37575)
0620 02:26:13.715936 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:13.715994 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:13.718321 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:13.719328 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:13.719344 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:13.720142 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:13.720760 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.720781 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.721907 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.721911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.723318 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.723331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.724265 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.724270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.724485 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.725266 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:13.725290 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:13.727699 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:13.731239 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:13.731251 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:13.731301 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:13.731979 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:13.732494 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:13.732500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:13.732503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:13.732778 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:13.733479 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:13.733486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:13.733863 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 02:26:13.734109 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:26:13.734589 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":375,"threads_started":1}
I20260620 02:26:13.737514 27854 heartbeater.cc:347] Connected to a master server at 127.9.231.126:38035
I20260620 02:26:13.737860 27854 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:13.738698 27854 heartbeater.cc:511] Master 127.9.231.126:38035 requested a full tablet report, sending...
I20260620 02:26:13.740484 27396 ts_manager.cc:194] Registered new tserver with Master: 359a6288eb3b488980a27d9385385fe5 (127.9.231.67:39641)
I20260620 02:26:13.741397 27396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:37575
I20260620 02:26:13.746026 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:26:13.765527 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.750988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.9.231.126:38035)
0620 02:26:13.751287 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.751296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.751715 (+ 419us) server_negotiation.cc:407] Connection header received
0620 02:26:13.751948 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.751954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.752034 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.752153 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:13.753823 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.754683 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.756253 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.756708 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.756849 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.759679 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:13.759684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:13.759687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:13.759728 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:13.762425 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:13.763325 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:13.763330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:13.763334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:13.763488 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:13.763914 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:13.763919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:13.763922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:13.764286 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:13.764562 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.764932 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.765198 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:26:13.770151 27396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:26:13.777519 27396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:26:13.805897 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.791596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.9.231.66:35967)
0620 02:26:13.794845 (+ 3249us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.794858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.794894 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:26:13.795009 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.795020 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.795327 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.795617 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:13.797036 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.799263 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.802834 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.804410 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.804545 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.804738 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.805391 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.805549 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3049,"thread_start_us":3289,"threads_started":1}
I20260620 02:26:13.805895 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.792415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.9.231.65:42597)
0620 02:26:13.793709 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.793723 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.794419 (+ 696us) server_negotiation.cc:407] Connection header received
0620 02:26:13.794769 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.794779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.795109 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.795331 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:13.796776 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.798994 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.802721 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.804084 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.804199 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.804475 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.805325 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.805511 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":824,"threads_started":1}
I20260620 02:26:13.806356 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.791580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:35967 (local address 127.0.0.1:53848)
0620 02:26:13.793938 (+ 2358us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:13.793996 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:13.794218 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:13.795932 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:13.795940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:13.795983 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:13.796752 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.796771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.799478 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.799487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.802617 (+ 3130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.802633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.804898 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.804904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.805164 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.805219 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 02:26:13.805317 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:26:13.805600 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2022,"mutex_wait_us":215,"thread_start_us":774,"threads_started":1}
I20260620 02:26:13.806825 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.791690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:42597 (local address 127.0.0.1:33738)
0620 02:26:13.794270 (+ 2580us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:13.794293 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:13.794553 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:13.795741 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:13.795752 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:13.795795 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:13.796513 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.796533 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.799329 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.799338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.802493 (+ 3155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.802506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.804662 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.804668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.804924 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.805054 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 02:26:13.806213 (+ 1159us) client_negotiation.cc:240] Negotiation successful
0620 02:26:13.806395 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2409,"mutex_wait_us":574,"spinlock_wait_cycles":1280,"thread_start_us":2231,"threads_started":1}
I20260620 02:26:13.810815 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.794561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:39641 (local address 127.0.0.1:56602)
0620 02:26:13.801428 (+ 6867us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:13.801457 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:13.801710 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:13.802493 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:13.802498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:13.802527 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:13.802955 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.802966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.804921 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.804926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.807525 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.807552 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.809932 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.809938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.810125 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.810160 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:26:13.810262 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:26:13.810401 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6724,"mutex_wait_us":3,"thread_start_us":6486,"threads_started":1}
I20260620 02:26:13.810824 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.794676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.9.231.67:39641)
0620 02:26:13.800874 (+ 6198us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.800885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.801573 (+ 688us) server_negotiation.cc:407] Connection header received
0620 02:26:13.801969 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.801979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.802263 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.802489 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:13.803178 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.804753 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.808028 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.809533 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.809889 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.810152 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.810315 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.810484 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6010,"thread_start_us":5825,"threads_started":1}
I20260620 02:26:13.810631 27655 tablet_service.cc:1467] Processing CreateTablet for tablet a44d45bd963e49369b9e01a55dfa6b6d (DEFAULT_TABLE table=test-table [id=695719042b3045a389093878c212a350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:13.811002 27521 tablet_service.cc:1467] Processing CreateTablet for tablet a44d45bd963e49369b9e01a55dfa6b6d (DEFAULT_TABLE table=test-table [id=695719042b3045a389093878c212a350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:13.812283 27655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44d45bd963e49369b9e01a55dfa6b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:13.812609 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44d45bd963e49369b9e01a55dfa6b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:13.815130 27789 tablet_service.cc:1467] Processing CreateTablet for tablet a44d45bd963e49369b9e01a55dfa6b6d (DEFAULT_TABLE table=test-table [id=695719042b3045a389093878c212a350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:13.816900 27789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44d45bd963e49369b9e01a55dfa6b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:13.826274 27873 tablet_bootstrap.cc:492] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Bootstrap starting.
I20260620 02:26:13.828706 27874 tablet_bootstrap.cc:492] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Bootstrap starting.
I20260620 02:26:13.830513 27873 tablet_bootstrap.cc:654] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:13.832044 27873 log.cc:824] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:13.832911 27874 tablet_bootstrap.cc:654] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:13.834035 27876 tablet_bootstrap.cc:492] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Bootstrap starting.
I20260620 02:26:13.834528 27874 log.cc:824] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:13.835952 27873 tablet_bootstrap.cc:492] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: No bootstrap required, opened a new log
I20260620 02:26:13.836279 27873 ts_tablet_manager.cc:1397] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 02:26:13.839377 27876 tablet_bootstrap.cc:654] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:13.840847 27876 log.cc:824] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:13.842785 27874 tablet_bootstrap.cc:492] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: No bootstrap required, opened a new log
I20260620 02:26:13.843182 27874 ts_tablet_manager.cc:1397] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260620 02:26:13.845144 27876 tablet_bootstrap.cc:492] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: No bootstrap required, opened a new log
I20260620 02:26:13.845440 27876 ts_tablet_manager.cc:1397] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 02:26:13.846575 27873 raft_consensus.cc:348] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.847152 27873 raft_consensus.cc:374] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:13.847270 27873 raft_consensus.cc:729] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b266fe402b4845b9222abe7fe253ff, State: Initialized, Role: FOLLOWER
I20260620 02:26:13.848124 27873 consensus_queue.cc:260] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.854158 27720 heartbeater.cc:503] Master 127.9.231.126:38035 was elected leader, sending a full tablet report...
I20260620 02:26:13.854632 27876 raft_consensus.cc:348] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.855149 27873 ts_tablet_manager.cc:1428] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260620 02:26:13.855280 27876 raft_consensus.cc:374] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:13.855379 27876 raft_consensus.cc:729] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359a6288eb3b488980a27d9385385fe5, State: Initialized, Role: FOLLOWER
I20260620 02:26:13.856196 27876 consensus_queue.cc:260] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.856241 27874 raft_consensus.cc:348] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.856726 27874 raft_consensus.cc:374] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:13.856884 27874 raft_consensus.cc:729] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ea89528d294d51975c8552e85f4f97, State: Initialized, Role: FOLLOWER
I20260620 02:26:13.857463 27854 heartbeater.cc:503] Master 127.9.231.126:38035 was elected leader, sending a full tablet report...
I20260620 02:26:13.857574 27874 consensus_queue.cc:260] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.858294 27876 ts_tablet_manager.cc:1428] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 02:26:13.859951 27874 ts_tablet_manager.cc:1428] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260620 02:26:13.891472 27879 raft_consensus.cc:482] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:26:13.891836 27879 raft_consensus.cc:504] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.893641 27879 leader_election.cc:288] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ea89528d294d51975c8552e85f4f97 (127.9.231.65:42597), 359a6288eb3b488980a27d9385385fe5 (127.9.231.67:39641)
I20260620 02:26:13.902549 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.895383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:36361 (local address 127.9.231.67:39641)
0620 02:26:13.895604 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.895610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.895924 (+ 314us) server_negotiation.cc:407] Connection header received
0620 02:26:13.896201 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.896206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.896286 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.896389 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:13.897139 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.898187 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.900299 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.901365 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.901518 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.901598 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.902167 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.902315 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:26:13.903255 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.894193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:37709 (local address 127.9.231.65:42597)
0620 02:26:13.894397 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.894402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.895186 (+ 784us) server_negotiation.cc:407] Connection header received
0620 02:26:13.895966 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.895975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.896080 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.896182 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:13.896885 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.898072 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.900827 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.902096 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.902243 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.902308 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.902821 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.902966 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:26:13.903394 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.895176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:39641 (local address 127.9.231.66:36361)
0620 02:26:13.895826 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:13.895850 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:13.896027 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:13.896451 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:13.896456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:13.896485 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:13.896976 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.896987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.898308 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.898313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.900103 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.900119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.901772 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.901778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.901956 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.901994 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:26:13.902792 (+ 798us) client_negotiation.cc:240] Negotiation successful
0620 02:26:13.902947 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":87,"thread_start_us":309,"threads_started":1}
I20260620 02:26:13.903988 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.894135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:42597 (local address 127.9.231.66:37709)
0620 02:26:13.895003 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:13.895025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:13.895224 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:13.896190 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:13.896194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:13.896218 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:13.896692 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.896730 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.898229 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.898235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.900629 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:13.900646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.902429 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.902435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:13.902631 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.902674 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:26:13.903484 (+ 810us) client_negotiation.cc:240] Negotiation successful
0620 02:26:13.903625 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":33,"thread_start_us":500,"threads_started":1}
I20260620 02:26:13.904541 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d45bd963e49369b9e01a55dfa6b6d" candidate_uuid: "30b266fe402b4845b9222abe7fe253ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359a6288eb3b488980a27d9385385fe5" is_pre_election: true
I20260620 02:26:13.905187 27809 raft_consensus.cc:2454] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b266fe402b4845b9222abe7fe253ff in term 0.
I20260620 02:26:13.905264 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d45bd963e49369b9e01a55dfa6b6d" candidate_uuid: "30b266fe402b4845b9222abe7fe253ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ea89528d294d51975c8552e85f4f97" is_pre_election: true
I20260620 02:26:13.905828 27541 raft_consensus.cc:2454] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b266fe402b4845b9222abe7fe253ff in term 0.
I20260620 02:26:13.905975 27610 leader_election.cc:302] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b266fe402b4845b9222abe7fe253ff, 359a6288eb3b488980a27d9385385fe5; no voters:
I20260620 02:26:13.906425 27879 raft_consensus.cc:2781] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:26:13.906629 27879 raft_consensus.cc:482] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:26:13.906764 27879 raft_consensus.cc:3037] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:13.909085 27879 raft_consensus.cc:504] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.909721 27879 leader_election.cc:288] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [CANDIDATE]: Term 1 election: Requested vote from peers b6ea89528d294d51975c8552e85f4f97 (127.9.231.65:42597), 359a6288eb3b488980a27d9385385fe5 (127.9.231.67:39641)
W20260620 02:26:13.910175 27721 tablet.cc:2369] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:13.910334 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d45bd963e49369b9e01a55dfa6b6d" candidate_uuid: "30b266fe402b4845b9222abe7fe253ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ea89528d294d51975c8552e85f4f97"
I20260620 02:26:13.910550 27541 raft_consensus.cc:3037] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:13.910568 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d45bd963e49369b9e01a55dfa6b6d" candidate_uuid: "30b266fe402b4845b9222abe7fe253ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359a6288eb3b488980a27d9385385fe5"
I20260620 02:26:13.910796 27809 raft_consensus.cc:3037] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:13.913216 27809 raft_consensus.cc:2454] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b266fe402b4845b9222abe7fe253ff in term 1.
I20260620 02:26:13.913744 27610 leader_election.cc:302] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b266fe402b4845b9222abe7fe253ff, 359a6288eb3b488980a27d9385385fe5; no voters:
I20260620 02:26:13.913867 27541 raft_consensus.cc:2454] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b266fe402b4845b9222abe7fe253ff in term 1.
I20260620 02:26:13.914052 27879 raft_consensus.cc:2781] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:13.914736 27879 raft_consensus.cc:686] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 1 LEADER]: Becoming Leader. State: Replica: 30b266fe402b4845b9222abe7fe253ff, State: Running, Role: LEADER
I20260620 02:26:13.915351 27879 consensus_queue.cc:237] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } }
I20260620 02:26:13.920750 27395 catalog_manager.cc:5495] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 30b266fe402b4845b9222abe7fe253ff (127.9.231.66). New cstate: current_term: 1 leader_uuid: "30b266fe402b4845b9222abe7fe253ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30b266fe402b4845b9222abe7fe253ff" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 35967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 } health_report { overall_health: UNKNOWN } } }
I20260620 02:26:13.940236 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.934201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.9.231.66:35967)
0620 02:26:13.934436 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.934449 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:13.934467 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:26:13.934609 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:13.934614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:13.934748 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:13.934884 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:26:13.935424 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.936395 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.937850 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:13.938281 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:13.938391 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:13.938906 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:26:13.939553 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:26:13.939695 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:13.939918 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 02:26:13.939999 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:26:13.949697 27541 raft_consensus.cc:1261] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 1 FOLLOWER]: Refusing update from remote peer 30b266fe402b4845b9222abe7fe253ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:13.949699 27809 raft_consensus.cc:1261] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 1 FOLLOWER]: Refusing update from remote peer 30b266fe402b4845b9222abe7fe253ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:13.950721 27879 consensus_queue.cc:1035] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "359a6288eb3b488980a27d9385385fe5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:13.951051 27884 consensus_queue.cc:1035] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ea89528d294d51975c8552e85f4f97" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:13.962703 27887 mvcc.cc:205] Tried to move back new op lower bound from 7298754043685781504 to 7298754043561963520. Current Snapshot: MvccSnapshot[applied={T|T < 7298754043685781504}]
W20260620 02:26:13.963339 27855 tablet.cc:2369] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:13.979254 27394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:13.979657 27394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:13.983599 27394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d327ac56eac4b2fadc65aa25c15445c: Sending DeleteTablet for 3 replicas of tablet a44d45bd963e49369b9e01a55dfa6b6d
I20260620 02:26:13.985019 27789 tablet_service.cc:1514] Processing DeleteTablet for tablet a44d45bd963e49369b9e01a55dfa6b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:13 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:56602
I20260620 02:26:13.985024 27655 tablet_service.cc:1514] Processing DeleteTablet for tablet a44d45bd963e49369b9e01a55dfa6b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:13 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:53848
I20260620 02:26:13.985435 27521 tablet_service.cc:1514] Processing DeleteTablet for tablet a44d45bd963e49369b9e01a55dfa6b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:13 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:33738
I20260620 02:26:13.986860 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27458
I20260620 02:26:13.997594 27896 tablet_replica.cc:330] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: stopping tablet replica
I20260620 02:26:13.998072 27896 raft_consensus.cc:2229] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:13.998606 27896 raft_consensus.cc:2258] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:14.001763 27896 ts_tablet_manager.cc:1905] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:14.002482 27897 tablet_replica.cc:330] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: stopping tablet replica
I20260620 02:26:14.003129 27897 raft_consensus.cc:2229] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:26:14.003950 27897 raft_consensus.cc:2258] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:14.006409 27897 ts_tablet_manager.cc:1905] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:14.009645 27896 ts_tablet_manager.cc:1918] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:14.009889 27896 log.cc:1196] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/a44d45bd963e49369b9e01a55dfa6b6d
I20260620 02:26:14.010569 27896 ts_tablet_manager.cc:1939] T a44d45bd963e49369b9e01a55dfa6b6d P 359a6288eb3b488980a27d9385385fe5: Deleting consensus metadata
I20260620 02:26:14.012765 27382 catalog_manager.cc:4841] TS 359a6288eb3b488980a27d9385385fe5 (127.9.231.67:39641): tablet a44d45bd963e49369b9e01a55dfa6b6d (table test-table [id=695719042b3045a389093878c212a350]) successfully deleted
I20260620 02:26:14.013713 27898 tablet_replica.cc:330] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: stopping tablet replica
I20260620 02:26:14.013715 27897 ts_tablet_manager.cc:1918] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:14.013902 27897 log.cc:1196] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/a44d45bd963e49369b9e01a55dfa6b6d
I20260620 02:26:14.014247 27898 raft_consensus.cc:2229] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:14.014518 27897 ts_tablet_manager.cc:1939] T a44d45bd963e49369b9e01a55dfa6b6d P 30b266fe402b4845b9222abe7fe253ff: Deleting consensus metadata
I20260620 02:26:14.014915 27898 raft_consensus.cc:2258] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:14.016494 27383 catalog_manager.cc:4841] TS 30b266fe402b4845b9222abe7fe253ff (127.9.231.66:35967): tablet a44d45bd963e49369b9e01a55dfa6b6d (table test-table [id=695719042b3045a389093878c212a350]) successfully deleted
I20260620 02:26:14.017863 27898 ts_tablet_manager.cc:1905] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:26:14 dist-test-slave-9vg1 krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:26:14.025789 27898 ts_tablet_manager.cc:1918] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:14.025996 27898 log.cc:1196] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/a44d45bd963e49369b9e01a55dfa6b6d
I20260620 02:26:14.026669 27898 ts_tablet_manager.cc:1939] T a44d45bd963e49369b9e01a55dfa6b6d P b6ea89528d294d51975c8552e85f4f97: Deleting consensus metadata
I20260620 02:26:14.028910 27382 catalog_manager.cc:4841] TS b6ea89528d294d51975c8552e85f4f97 (127.9.231.65:42597): tablet a44d45bd963e49369b9e01a55dfa6b6d (table test-table [id=695719042b3045a389093878c212a350]) successfully deleted
I20260620 02:26:14.030105 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:13.991258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.9.231.65:42597)
0620 02:26:13.991508 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:13.991517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:14.009642 (+ 18125us) server_negotiation.cc:407] Connection header received
0620 02:26:14.010343 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:14.010378 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:14.010508 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:14.010630 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:14.014211 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:14.015465 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:14.017466 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:14.017902 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:14.018060 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:14.022875 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:14.022889 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:14.022906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:14.022965 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:14.026552 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:14.027587 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:14.027596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:14.027598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:14.027767 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:14.028223 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:14.028228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:14.028230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:14.028600 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:14.028917 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:14.029500 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 02:26:14.029746 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:26:14.030843 27581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:14.124750 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27458
I20260620 02:26:14.145200 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27592
Jun 20 02:26:14 dist-test-slave-9vg1 krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:26:14.166193 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:14.146167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.9.231.66:35967)
0620 02:26:14.146340 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:14.146344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:14.146362 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:26:14.146450 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:14.146454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:14.146538 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:14.146670 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:14.148478 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:14.149332 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:14.151444 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:14.151932 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:14.152054 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:14.157990 (+ 5936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:14.158006 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:14.158029 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:14.158095 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:14.162586 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:14.163642 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:14.163656 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:14.163662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:14.163862 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:14.164322 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:14.164328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:14.164331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:14.164849 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:14.165205 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:14.165531 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:26:14.165783 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:26:14.167012 27715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:14.286815 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27592
I20260620 02:26:14.307015 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27726
Jun 20 02:26:14 dist-test-slave-9vg1 krb5kdc[27351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:26:14.324405 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:14.308046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.9.231.67:39641)
0620 02:26:14.308236 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:14.308240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:14.308259 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:26:14.308410 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:14.308414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:14.308489 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:14.308595 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:14.309917 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:14.310633 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:14.312264 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:14.312659 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:14.312757 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:14.317815 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:14.317833 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:14.317851 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:14.317895 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:14.321090 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:14.322244 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:14.322252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:14.322255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:14.322400 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:14.322806 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:14.322809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:14.322811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:14.323298 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:14.323517 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:14.323931 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 02:26:14.324123 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:26:14.325177 27849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:14.417068 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27726
I20260620 02:26:14.435608 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27364
I20260620 02:26:14.436908 27426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:14.554128 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27364
2026-06-20T02:26:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4568 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T02:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:14Z Disabled control of system clock
I20260620 02:26:14.639477 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:42591
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:26:15.011582 27907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:15.011902 27907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:15.011987 27907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:15.019470 27907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:15.019608 27907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:15.019649 27907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:15.019709 27907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:15.019757 27907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:15.029767 27907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:42591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:15.031180 27907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:15.032984 27907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:15.045961 27915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:15.046108 27912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:15.046054 27913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:15.047672 27907 server_base.cc:1034] running on GCE node
I20260620 02:26:15.048581 27907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:15.050232 27907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:15.051479 27907 hybrid_clock.cc:648] HybridClock initialized: now 1781922375051433 us; error 77 us; skew 500 ppm
I20260620 02:26:15.051916 27907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:15.054471 27907 webserver.cc:460] Webserver started at http://127.9.231.126:33023/ using document root <none> and password file <none>
I20260620 02:26:15.055343 27907 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:15.055480 27907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:15.055816 27907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:15.058919 27907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "0551cc5bd76944e9b54026c114f964b0"
format_stamp: "Formatted at 2026-06-20 02:26:15 on dist-test-slave-9vg1"
server_key: "db83658306927fd7267a27fefe5510a9"
server_key_iv: "c02bf5f5744801d08dc9c6bf3e30d7a3"
server_key_version: "encryptionkey@0"
I20260620 02:26:15.059899 27907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "0551cc5bd76944e9b54026c114f964b0"
format_stamp: "Formatted at 2026-06-20 02:26:15 on dist-test-slave-9vg1"
server_key: "db83658306927fd7267a27fefe5510a9"
server_key_iv: "c02bf5f5744801d08dc9c6bf3e30d7a3"
server_key_version: "encryptionkey@0"
I20260620 02:26:15.066140 27907 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 02:26:15.070199 27921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:15.071866 27907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:26:15.072094 27907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "0551cc5bd76944e9b54026c114f964b0"
format_stamp: "Formatted at 2026-06-20 02:26:15 on dist-test-slave-9vg1"
server_key: "db83658306927fd7267a27fefe5510a9"
server_key_iv: "c02bf5f5744801d08dc9c6bf3e30d7a3"
server_key_version: "encryptionkey@0"
I20260620 02:26:15.072357 27907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:15.084098 27907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:15.085125 27907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:15.085451 27907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:15.106235 27907 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:42591
I20260620 02:26:15.106257 27972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:42591 every 8 connection(s)
I20260620 02:26:15.107760 27907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:15.111811 27973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:15.116256 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27907
I20260620 02:26:15.116453 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:15.116920 10141 external_mini_cluster.cc:1428] Setting key f1a94fa92cb855fd0c500dd4d47f3a83
I20260620 02:26:15.125378 27973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0: Bootstrap starting.
I20260620 02:26:15.129720 27973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:15.131155 27973 log.cc:824] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:15.134241 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:15.118880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.9.231.126:42591)
0620 02:26:15.120154 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:15.120172 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:15.120224 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:26:15.123304 (+ 3080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:15.123340 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:15.124020 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:15.124484 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:15.125065 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:15.127316 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:15.128890 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:15.130067 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:15.130317 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:15.131600 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:15.131613 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:15.131635 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:15.132364 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:15.132472 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:15.132497 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:15.132651 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:15.132902 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 02:26:15.133259 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":453,"threads_started":1}
I20260620 02:26:15.134537 27973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0: No bootstrap required, opened a new log
I20260620 02:26:15.142966 27973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0551cc5bd76944e9b54026c114f964b0" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42591 } }
I20260620 02:26:15.143589 27973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:15.143759 27973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0551cc5bd76944e9b54026c114f964b0, State: Initialized, Role: FOLLOWER
I20260620 02:26:15.144445 27973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0551cc5bd76944e9b54026c114f964b0" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42591 } }
I20260620 02:26:15.144775 27973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:15.144938 27973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:15.145134 27973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:15.147862 27973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0551cc5bd76944e9b54026c114f964b0" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42591 } }
I20260620 02:26:15.148526 27973 leader_election.cc:302] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0551cc5bd76944e9b54026c114f964b0; no voters:
I20260620 02:26:15.149103 27973 leader_election.cc:288] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:15.149261 27978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:15.150008 27978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0551cc5bd76944e9b54026c114f964b0, State: Running, Role: LEADER
I20260620 02:26:15.150611 27978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0551cc5bd76944e9b54026c114f964b0" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42591 } }
I20260620 02:26:15.151358 27973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:15.153805 27979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0551cc5bd76944e9b54026c114f964b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0551cc5bd76944e9b54026c114f964b0" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42591 } } }
I20260620 02:26:15.153803 27980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0551cc5bd76944e9b54026c114f964b0. Latest consensus state: current_term: 1 leader_uuid: "0551cc5bd76944e9b54026c114f964b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0551cc5bd76944e9b54026c114f964b0" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42591 } } }
I20260620 02:26:15.154337 27979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:15.154407 27980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:15.157058 27986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:15.162165 27986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:15.171459 27986 catalog_manager.cc:1347] Generated new cluster ID: 4b554390248f48808123bd7eee81719d
I20260620 02:26:15.171594 27986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:15.180136 27986 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:15.182121 27986 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:15.202843 27986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0: Generated new TSK 0
I20260620 02:26:15.203878 27986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:15.220293 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42591
--builtin_ntp_servers=127.9.231.84:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:26:15.589835 27997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:15.590132 27997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:15.590224 27997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:15.597864 27997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:15.597986 27997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:15.598096 27997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:26:15.608083 27997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:15.609388 27997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:15.611231 27997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:15.625592 28005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:15.625803 28002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:15.625782 28003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:15.626441 27997 server_base.cc:1034] running on GCE node
I20260620 02:26:15.627243 27997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:15.628386 27997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:15.629673 27997 hybrid_clock.cc:648] HybridClock initialized: now 1781922375629594 us; error 61 us; skew 500 ppm
I20260620 02:26:15.630048 27997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:15.632992 27997 webserver.cc:460] Webserver started at http://127.9.231.65:34047/ using document root <none> and password file <none>
I20260620 02:26:15.633891 27997 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:15.634042 27997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:15.634403 27997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:15.637825 27997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "7825c73034464e4182195e113c15c8e6"
format_stamp: "Formatted at 2026-06-20 02:26:15 on dist-test-slave-9vg1"
server_key: "bb4864914b096d8b1a49388dd6f460a7"
server_key_iv: "2b2591c0eea340909b890edb6af40b3d"
server_key_version: "encryptionkey@0"
I20260620 02:26:15.638743 27997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "7825c73034464e4182195e113c15c8e6"
format_stamp: "Formatted at 2026-06-20 02:26:15 on dist-test-slave-9vg1"
server_key: "bb4864914b096d8b1a49388dd6f460a7"
server_key_iv: "2b2591c0eea340909b890edb6af40b3d"
server_key_version: "encryptionkey@0"
I20260620 02:26:15.644882 27997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 02:26:15.649304 28011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:15.650976 27997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:26:15.651247 27997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "7825c73034464e4182195e113c15c8e6"
format_stamp: "Formatted at 2026-06-20 02:26:15 on dist-test-slave-9vg1"
server_key: "bb4864914b096d8b1a49388dd6f460a7"
server_key_iv: "2b2591c0eea340909b890edb6af40b3d"
server_key_version: "encryptionkey@0"
I20260620 02:26:15.651539 27997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:15.670286 27997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:15.671301 27997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:15.671590 27997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:15.672665 27997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:15.674358 27997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:15.674494 27997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:15.674582 27997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:15.674641 27997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:15.713965 27997 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:43561
I20260620 02:26:15.714017 28123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:43561 every 8 connection(s)
I20260620 02:26:15.715799 27997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:26:15.718217 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 27997
I20260620 02:26:15.718438 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:26:15.718993 10141 external_mini_cluster.cc:1428] Setting key 91624ebb612347a1306312a7fcde4a8d
I20260620 02:26:15.722229 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42591
--builtin_ntp_servers=127.9.231.84:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:26:15.731222 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:15.719120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:40651 (local address 127.9.231.126:42591)
0620 02:26:15.719670 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:15.719675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:15.721301 (+ 1626us) server_negotiation.cc:407] Connection header received
0620 02:26:15.723464 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:15.723469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:15.723581 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:15.723711 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:15.724952 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:15.725695 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:15.727488 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:15.728032 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:15.728192 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:15.729912 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:15.729918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:15.729921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:15.729943 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:15.729988 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:15.730002 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:15.730099 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:15.730411 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 02:26:15.730507 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":242,"threads_started":1}
I20260620 02:26:15.732394 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:15.719889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:42591 (local address 127.9.231.65:40651)
0620 02:26:15.721146 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:15.721200 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:15.723121 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:15.723986 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:15.723995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:15.724107 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:15.724757 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:15.724776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:15.725891 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:15.725895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:15.727309 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:15.727324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:15.728342 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:15.728353 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:15.728604 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:15.729443 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:15.729448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:15.729515 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:15.729639 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:15.730127 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:15.730139 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:15.730157 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 02:26:15.730379 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 02:26:15.730865 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":421,"threads_started":1}
I20260620 02:26:15.734280 28124 heartbeater.cc:347] Connected to a master server at 127.9.231.126:42591
I20260620 02:26:15.734676 28124 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:15.735602 28124 heartbeater.cc:511] Master 127.9.231.126:42591 requested a full tablet report, sending...
I20260620 02:26:15.738044 27938 ts_manager.cc:194] Registered new tserver with Master: 7825c73034464e4182195e113c15c8e6 (127.9.231.65:43561)
I20260620 02:26:15.739748 27938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.65:40651
W20260620 02:26:16.071754 28128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:16.072062 28128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:16.072213 28128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:16.080058 28128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:16.080192 28128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:16.080365 28128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:26:16.094204 28128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:16.095743 28128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:16.097642 28128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:16.113960 28134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:16.114168 28133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:16.114233 28136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:16.116236 28128 server_base.cc:1034] running on GCE node
I20260620 02:26:16.116930 28128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:16.118042 28128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:16.119279 28128 hybrid_clock.cc:648] HybridClock initialized: now 1781922376119218 us; error 49 us; skew 500 ppm
I20260620 02:26:16.119680 28128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:16.122283 28128 webserver.cc:460] Webserver started at http://127.9.231.66:37431/ using document root <none> and password file <none>
I20260620 02:26:16.123039 28128 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:16.123263 28128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:16.123608 28128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:16.126783 28128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "dbede4f033d9402ca7c9cf6c288c98dc"
format_stamp: "Formatted at 2026-06-20 02:26:16 on dist-test-slave-9vg1"
server_key: "ba0da06ac1e30a44d45ca348aea98562"
server_key_iv: "330e57386f66e37be2bcf623be6c0521"
server_key_version: "encryptionkey@0"
I20260620 02:26:16.127715 28128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "dbede4f033d9402ca7c9cf6c288c98dc"
format_stamp: "Formatted at 2026-06-20 02:26:16 on dist-test-slave-9vg1"
server_key: "ba0da06ac1e30a44d45ca348aea98562"
server_key_iv: "330e57386f66e37be2bcf623be6c0521"
server_key_version: "encryptionkey@0"
I20260620 02:26:16.133900 28128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 02:26:16.137975 28142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:16.139562 28128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:16.139783 28128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "dbede4f033d9402ca7c9cf6c288c98dc"
format_stamp: "Formatted at 2026-06-20 02:26:16 on dist-test-slave-9vg1"
server_key: "ba0da06ac1e30a44d45ca348aea98562"
server_key_iv: "330e57386f66e37be2bcf623be6c0521"
server_key_version: "encryptionkey@0"
I20260620 02:26:16.140065 28128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:16.169328 28128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:16.170266 28128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:16.170599 28128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:16.171648 28128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:16.173226 28128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:16.173359 28128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:16.173483 28128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:16.173573 28128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:16.207998 28128 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:42827
I20260620 02:26:16.208091 28254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:42827 every 8 connection(s)
I20260620 02:26:16.209381 28128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:26:16.211757 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 28128
I20260620 02:26:16.211941 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:26:16.212487 10141 external_mini_cluster.cc:1428] Setting key 90278a40ebc9206efe7689628483af48
I20260620 02:26:16.215406 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42591
--builtin_ntp_servers=127.9.231.84:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:26:16.223156 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.212455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:55733 (local address 127.9.231.126:42591)
0620 02:26:16.212677 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.212684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.214331 (+ 1647us) server_negotiation.cc:407] Connection header received
0620 02:26:16.216405 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.216414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.216518 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.216634 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:16.217783 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.218517 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.220254 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.220667 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.220807 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.222275 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:16.222280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:16.222283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:16.222304 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:16.222337 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:16.222349 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:16.222445 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.222686 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.222792 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:26:16.224237 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.213053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:42591 (local address 127.9.231.66:55733)
0620 02:26:16.214173 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:16.214218 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:16.216086 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:16.216852 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:16.216862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:16.216959 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:16.217572 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.217610 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.218678 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.218686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.220089 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.220103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.220916 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.220925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.221165 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.221904 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:16.221907 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:16.221949 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:16.222038 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:16.222458 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:16.222472 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:16.222480 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:26:16.222669 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 02:26:16.223123 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":370,"threads_started":1}
I20260620 02:26:16.225859 28255 heartbeater.cc:347] Connected to a master server at 127.9.231.126:42591
I20260620 02:26:16.226214 28255 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:16.227200 28255 heartbeater.cc:511] Master 127.9.231.126:42591 requested a full tablet report, sending...
I20260620 02:26:16.228839 27938 ts_manager.cc:194] Registered new tserver with Master: dbede4f033d9402ca7c9cf6c288c98dc (127.9.231.66:42827)
I20260620 02:26:16.229753 27938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.66:55733
W20260620 02:26:16.579806 28258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:16.580154 28258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:16.580376 28258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:16.589092 28258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:16.589231 28258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:16.589437 28258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:26:16.600790 28258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:16.602434 28258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:16.604578 28258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:16.620078 28264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:16.620152 28263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:16.620128 28266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:16.624972 28258 server_base.cc:1034] running on GCE node
I20260620 02:26:16.625777 28258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:16.626957 28258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:16.628240 28258 hybrid_clock.cc:648] HybridClock initialized: now 1781922376628194 us; error 44 us; skew 500 ppm
I20260620 02:26:16.628624 28258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:16.631538 28258 webserver.cc:460] Webserver started at http://127.9.231.67:38049/ using document root <none> and password file <none>
I20260620 02:26:16.632378 28258 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:16.632534 28258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:16.632926 28258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:16.636413 28258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "5263165190ff4097afd25e53c66e11f9"
format_stamp: "Formatted at 2026-06-20 02:26:16 on dist-test-slave-9vg1"
server_key: "d8a65ce7d438e53fda2ce42dc1a51504"
server_key_iv: "5041ad79aa4a666b75eef9593b05c299"
server_key_version: "encryptionkey@0"
I20260620 02:26:16.637383 28258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "5263165190ff4097afd25e53c66e11f9"
format_stamp: "Formatted at 2026-06-20 02:26:16 on dist-test-slave-9vg1"
server_key: "d8a65ce7d438e53fda2ce42dc1a51504"
server_key_iv: "5041ad79aa4a666b75eef9593b05c299"
server_key_version: "encryptionkey@0"
I20260620 02:26:16.643980 28258 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:26:16.648468 28272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:16.650142 28258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:26:16.650426 28258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "5263165190ff4097afd25e53c66e11f9"
format_stamp: "Formatted at 2026-06-20 02:26:16 on dist-test-slave-9vg1"
server_key: "d8a65ce7d438e53fda2ce42dc1a51504"
server_key_iv: "5041ad79aa4a666b75eef9593b05c299"
server_key_version: "encryptionkey@0"
I20260620 02:26:16.650713 28258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:16.682134 28258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:16.683218 28258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:16.683564 28258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:16.684655 28258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:16.686458 28258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:16.686596 28258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:16.686743 28258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:16.686831 28258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:16.722857 28258 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:46439
I20260620 02:26:16.722916 28384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:46439 every 8 connection(s)
I20260620 02:26:16.724311 28258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:26:16.726686 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 28258
I20260620 02:26:16.726881 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:26:16.727514 10141 external_mini_cluster.cc:1428] Setting key f28c76cdfe12cf15f006ce07eb8f3f2e
I20260620 02:26:16.737993 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.727381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:33593 (local address 127.9.231.126:42591)
0620 02:26:16.727939 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.727964 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.729357 (+ 1393us) server_negotiation.cc:407] Connection header received
0620 02:26:16.731451 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.731455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.731532 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.731639 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:16.732785 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.733568 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.735278 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.735724 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.735832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.737197 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:16.737202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:16.737205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:16.737225 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:16.737262 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:16.737275 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:16.737344 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.737597 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.737687 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":340,"threads_started":1}
I20260620 02:26:16.739024 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.728007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:42591 (local address 127.9.231.67:33593)
0620 02:26:16.729203 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:16.729252 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:16.731134 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:16.731886 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:16.731895 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:16.731994 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:16.732606 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.732624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.733762 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.733767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.735137 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.735151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.735929 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.735934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.736152 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.736872 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:16.736875 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:16.736922 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:16.737004 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:16.737359 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:16.737366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:16.737372 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:26:16.737529 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 02:26:16.737883 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":369,"threads_started":1}
I20260620 02:26:16.740759 28385 heartbeater.cc:347] Connected to a master server at 127.9.231.126:42591
I20260620 02:26:16.741199 28385 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:16.742249 28385 heartbeater.cc:511] Master 127.9.231.126:42591 requested a full tablet report, sending...
I20260620 02:26:16.743755 28124 heartbeater.cc:503] Master 127.9.231.126:42591 was elected leader, sending a full tablet report...
I20260620 02:26:16.744202 27938 ts_manager.cc:194] Registered new tserver with Master: 5263165190ff4097afd25e53c66e11f9 (127.9.231.67:46439)
I20260620 02:26:16.745528 27938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.67:33593
I20260620 02:26:16.756831 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:26:16.767421 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.761168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.9.231.126:42591)
0620 02:26:16.761438 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.761444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.761701 (+ 257us) server_negotiation.cc:407] Connection header received
0620 02:26:16.761832 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.761836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.761902 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.761985 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:16.762480 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.763315 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.764796 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.765208 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.765312 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.766538 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:16.766546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:16.766550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:16.766575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:16.766612 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:16.766637 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:16.766723 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.766941 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.767028 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 02:26:16.771992 27938 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:26:16.779496 27938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:26:16.816496 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.793901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.9.231.67:46439)
0620 02:26:16.807399 (+ 13498us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.807411 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.807458 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:26:16.807590 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.807599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.807933 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.808879 (+ 946us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:16.809203 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.810693 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.813828 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.815151 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.815285 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.815596 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.815876 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.816085 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13267,"thread_start_us":13057,"threads_started":1}
I20260620 02:26:16.816425 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.793776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:46439 (local address 127.0.0.1:35842)
0620 02:26:16.806437 (+ 12661us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:16.806481 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:16.806628 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:16.808287 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:16.808299 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:16.808348 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:16.808933 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.808952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.810906 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.810911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.813624 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.813637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.815364 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.815371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.815596 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.815655 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 02:26:16.815802 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 02:26:16.816029 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12477,"mutex_wait_us":97,"thread_start_us":14743,"threads_started":1}
I20260620 02:26:16.817559 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.793753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:42827 (local address 127.0.0.1:54874)
0620 02:26:16.804064 (+ 10311us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:16.804149 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:16.804409 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:16.809062 (+ 4653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:16.809067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:16.809090 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:16.809454 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.809466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.811612 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.811618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.814137 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.814168 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.816089 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.816095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.816271 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.816313 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:26:16.816392 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:26:16.816549 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9934,"mutex_wait_us":165,"thread_start_us":10811,"threads_started":1}
I20260620 02:26:16.817731 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.796130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.9.231.65:43561)
0620 02:26:16.807927 (+ 11797us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.807939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.809193 (+ 1254us) server_negotiation.cc:407] Connection header received
0620 02:26:16.809305 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.809315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.809550 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.809770 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:16.810422 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.812056 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.814668 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.816031 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.816163 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.816437 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.817134 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.817351 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11554,"thread_start_us":11381,"threads_started":1}
I20260620 02:26:16.818164 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.794456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.9.231.66:42827)
0620 02:26:16.808419 (+ 13963us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.808429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.808463 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:26:16.808591 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.808600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.808833 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.809683 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:16.809965 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.811427 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.814385 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.815828 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.816686 (+ 858us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.816923 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.817011 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.817200 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13788,"thread_start_us":13658,"threads_started":1}
I20260620 02:26:16.818288 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.795478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57070)
0620 02:26:16.808873 (+ 13395us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:16.808908 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:16.809059 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:16.809803 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:16.809808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:16.809834 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:16.810208 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.810220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.812216 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.812222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.814492 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.814503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.816586 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.816592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.816765 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.816794 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:26:16.816882 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:26:16.817027 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13240,"mutex_wait_us":54,"thread_start_us":7997,"threads_started":1}
I20260620 02:26:16.822044 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 39dafdc38f6c4746944736134f9929bd (DEFAULT_TABLE table=test-table [id=5b956bdf27e34960bdd4ae3ad9871fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:16.822057 28059 tablet_service.cc:1467] Processing CreateTablet for tablet 39dafdc38f6c4746944736134f9929bd (DEFAULT_TABLE table=test-table [id=5b956bdf27e34960bdd4ae3ad9871fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:16.822057 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 39dafdc38f6c4746944736134f9929bd (DEFAULT_TABLE table=test-table [id=5b956bdf27e34960bdd4ae3ad9871fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:16.823531 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39dafdc38f6c4746944736134f9929bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:16.823552 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39dafdc38f6c4746944736134f9929bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:16.823544 28059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39dafdc38f6c4746944736134f9929bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:16.836061 28404 tablet_bootstrap.cc:492] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Bootstrap starting.
I20260620 02:26:16.839023 28405 tablet_bootstrap.cc:492] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Bootstrap starting.
I20260620 02:26:16.839958 28406 tablet_bootstrap.cc:492] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: Bootstrap starting.
I20260620 02:26:16.840721 28404 tablet_bootstrap.cc:654] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:16.842139 28404 log.cc:824] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:16.843318 28405 tablet_bootstrap.cc:654] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:16.844336 28406 tablet_bootstrap.cc:654] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:16.844969 28405 log.cc:824] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:16.845611 28406 log.cc:824] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:16.847038 28404 tablet_bootstrap.cc:492] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: No bootstrap required, opened a new log
I20260620 02:26:16.847433 28404 ts_tablet_manager.cc:1397] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260620 02:26:16.848388 28406 tablet_bootstrap.cc:492] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: No bootstrap required, opened a new log
I20260620 02:26:16.848685 28406 ts_tablet_manager.cc:1397] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:26:16.848941 28405 tablet_bootstrap.cc:492] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: No bootstrap required, opened a new log
I20260620 02:26:16.849241 28405 ts_tablet_manager.cc:1397] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 02:26:16.857265 28406 raft_consensus.cc:348] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.857751 28406 raft_consensus.cc:374] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:16.857853 28406 raft_consensus.cc:729] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7825c73034464e4182195e113c15c8e6, State: Initialized, Role: FOLLOWER
I20260620 02:26:16.858754 28406 consensus_queue.cc:260] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.859297 28404 raft_consensus.cc:348] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.859756 28404 raft_consensus.cc:374] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:16.859844 28404 raft_consensus.cc:729] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5263165190ff4097afd25e53c66e11f9, State: Initialized, Role: FOLLOWER
I20260620 02:26:16.860464 28404 consensus_queue.cc:260] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.860493 28405 raft_consensus.cc:348] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.860896 28405 raft_consensus.cc:374] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:16.861006 28405 raft_consensus.cc:729] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbede4f033d9402ca7c9cf6c288c98dc, State: Initialized, Role: FOLLOWER
I20260620 02:26:16.861855 28405 consensus_queue.cc:260] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.867722 28405 ts_tablet_manager.cc:1428] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
I20260620 02:26:16.868265 28255 heartbeater.cc:503] Master 127.9.231.126:42591 was elected leader, sending a full tablet report...
I20260620 02:26:16.871253 28385 heartbeater.cc:503] Master 127.9.231.126:42591 was elected leader, sending a full tablet report...
I20260620 02:26:16.871886 28404 ts_tablet_manager.cc:1428] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Time spent starting tablet: real 0.024s user 0.012s sys 0.012s
I20260620 02:26:16.875339 28406 ts_tablet_manager.cc:1428] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: Time spent starting tablet: real 0.026s user 0.016s sys 0.010s
W20260620 02:26:16.961592 28256 tablet.cc:2369] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:16.965338 28412 raft_consensus.cc:482] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:26:16.965643 28412 raft_consensus.cc:504] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.967393 28412 leader_election.cc:288] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5263165190ff4097afd25e53c66e11f9 (127.9.231.67:46439), 7825c73034464e4182195e113c15c8e6 (127.9.231.65:43561)
W20260620 02:26:16.968391 28125 tablet.cc:2369] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:26:16.976101 28386 tablet.cc:2369] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:16.977562 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.967844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:36013 (local address 127.9.231.65:43561)
0620 02:26:16.968064 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.968071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.969611 (+ 1540us) server_negotiation.cc:407] Connection header received
0620 02:26:16.969810 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.969816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.969900 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.970001 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:16.970841 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.972023 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.975112 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.976395 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.976588 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.976720 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.977156 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.977310 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:26:16.977726 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.967713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:46439 (local address 127.9.231.66:45977)
0620 02:26:16.969506 (+ 1793us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:16.969524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:16.969672 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:16.970031 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:16.970037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:16.970083 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:16.970697 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.970710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.972179 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.972185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.974806 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.974825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.976905 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.976911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.977087 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.977129 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:26:16.977233 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:26:16.977359 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1697,"mutex_wait_us":134,"spinlock_wait_cycles":65664,"thread_start_us":1514,"threads_started":1}
I20260620 02:26:16.977861 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.967918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:45977 (local address 127.9.231.67:46439)
0620 02:26:16.968185 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:16.968195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:16.969610 (+ 1415us) server_negotiation.cc:407] Connection header received
0620 02:26:16.969810 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:16.969816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:16.969900 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:16.970001 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:16.970862 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.971986 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.975100 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.976520 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.976679 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.976748 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:16.977268 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 02:26:16.977477 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:26:16.977994 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:16.967683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.9.231.66:36013)
0620 02:26:16.969473 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:16.969499 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:16.969660 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:16.970031 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:16.970037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:16.970080 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:16.970628 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.970642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.972179 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.972185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.974810 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:16.974825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:16.976617 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:16.976627 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:16.976921 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:16.976982 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 02:26:16.977099 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:26:16.977269 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"mutex_wait_us":166,"spinlock_wait_cycles":13440,"thread_start_us":522,"threads_started":1}
I20260620 02:26:16.978931 28079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39dafdc38f6c4746944736134f9929bd" candidate_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7825c73034464e4182195e113c15c8e6" is_pre_election: true
I20260620 02:26:16.978925 28340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39dafdc38f6c4746944736134f9929bd" candidate_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5263165190ff4097afd25e53c66e11f9" is_pre_election: true
I20260620 02:26:16.979563 28340 raft_consensus.cc:2454] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbede4f033d9402ca7c9cf6c288c98dc in term 0.
I20260620 02:26:16.979563 28079 raft_consensus.cc:2454] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbede4f033d9402ca7c9cf6c288c98dc in term 0.
I20260620 02:26:16.980288 28144 leader_election.cc:302] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7825c73034464e4182195e113c15c8e6, dbede4f033d9402ca7c9cf6c288c98dc; no voters:
I20260620 02:26:16.980777 28412 raft_consensus.cc:2781] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:26:16.980957 28412 raft_consensus.cc:482] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:26:16.981067 28412 raft_consensus.cc:3037] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:16.983574 28412 raft_consensus.cc:504] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.984256 28412 leader_election.cc:288] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [CANDIDATE]: Term 1 election: Requested vote from peers 5263165190ff4097afd25e53c66e11f9 (127.9.231.67:46439), 7825c73034464e4182195e113c15c8e6 (127.9.231.65:43561)
I20260620 02:26:16.984824 28340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39dafdc38f6c4746944736134f9929bd" candidate_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5263165190ff4097afd25e53c66e11f9"
I20260620 02:26:16.984844 28079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39dafdc38f6c4746944736134f9929bd" candidate_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7825c73034464e4182195e113c15c8e6"
I20260620 02:26:16.985046 28340 raft_consensus.cc:3037] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:16.985060 28079 raft_consensus.cc:3037] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:16.987409 28340 raft_consensus.cc:2454] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbede4f033d9402ca7c9cf6c288c98dc in term 1.
I20260620 02:26:16.987429 28079 raft_consensus.cc:2454] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbede4f033d9402ca7c9cf6c288c98dc in term 1.
I20260620 02:26:16.987982 28145 leader_election.cc:302] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5263165190ff4097afd25e53c66e11f9, 7825c73034464e4182195e113c15c8e6, dbede4f033d9402ca7c9cf6c288c98dc; no voters:
I20260620 02:26:16.988345 28412 raft_consensus.cc:2781] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:16.988844 28412 raft_consensus.cc:686] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 1 LEADER]: Becoming Leader. State: Replica: dbede4f033d9402ca7c9cf6c288c98dc, State: Running, Role: LEADER
I20260620 02:26:16.989342 28412 consensus_queue.cc:237] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } }
I20260620 02:26:16.994567 27936 catalog_manager.cc:5495] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc reported cstate change: term changed from 0 to 1, leader changed from <none> to dbede4f033d9402ca7c9cf6c288c98dc (127.9.231.66). New cstate: current_term: 1 leader_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbede4f033d9402ca7c9cf6c288c98dc" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 42827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20260620 02:26:17.057011 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.051379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.9.231.66:42827)
0620 02:26:17.051577 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:17.051582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:17.051599 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:26:17.051720 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:17.051724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:17.051817 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:17.052014 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:26:17.052478 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.053203 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.054597 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.055162 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.055269 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:17.055793 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:26:17.056507 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:26:17.056673 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:17.056734 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 02:26:17.056802 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:26:17.065464 28340 raft_consensus.cc:1261] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 1 FOLLOWER]: Refusing update from remote peer dbede4f033d9402ca7c9cf6c288c98dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:17.065461 28079 raft_consensus.cc:1261] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 1 FOLLOWER]: Refusing update from remote peer dbede4f033d9402ca7c9cf6c288c98dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:17.066491 28415 consensus_queue.cc:1035] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7825c73034464e4182195e113c15c8e6" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:17.066861 28412 consensus_queue.cc:1035] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5263165190ff4097afd25e53c66e11f9" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:17.077900 28418 mvcc.cc:205] Tried to move back new op lower bound from 7298754056449765376 to 7298754056152158208. Current Snapshot: MvccSnapshot[applied={T|T < 7298754056449765376}]
I20260620 02:26:17.084118 28419 mvcc.cc:205] Tried to move back new op lower bound from 7298754056449765376 to 7298754056152158208. Current Snapshot: MvccSnapshot[applied={T|T < 7298754056449765376}]
I20260620 02:26:17.084481 28420 mvcc.cc:205] Tried to move back new op lower bound from 7298754056449765376 to 7298754056152158208. Current Snapshot: MvccSnapshot[applied={T|T < 7298754056449765376}]
I20260620 02:26:17.093449 27936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:17.093767 27936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:17.097819 27936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0551cc5bd76944e9b54026c114f964b0: Sending DeleteTablet for 3 replicas of tablet 39dafdc38f6c4746944736134f9929bd
I20260620 02:26:17.099241 28320 tablet_service.cc:1514] Processing DeleteTablet for tablet 39dafdc38f6c4746944736134f9929bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:17 UTC) from {username='slave'} at 127.0.0.1:35842
I20260620 02:26:17.099246 28059 tablet_service.cc:1514] Processing DeleteTablet for tablet 39dafdc38f6c4746944736134f9929bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:17 UTC) from {username='slave'} at 127.0.0.1:57070
I20260620 02:26:17.099614 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 39dafdc38f6c4746944736134f9929bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:17 UTC) from {username='slave'} at 127.0.0.1:54874
I20260620 02:26:17.100806 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27997
I20260620 02:26:17.111634 28429 tablet_replica.cc:330] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: stopping tablet replica
I20260620 02:26:17.112254 28429 raft_consensus.cc:2229] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:26:17.112555 28428 tablet_replica.cc:330] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: stopping tablet replica
I20260620 02:26:17.112963 28428 raft_consensus.cc:2229] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:17.113154 28429 raft_consensus.cc:2258] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:17.113461 28428 raft_consensus.cc:2258] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:17.115598 28429 ts_tablet_manager.cc:1905] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:17.119405 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.102307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.9.231.65:43561)
0620 02:26:17.102492 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:17.102498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:17.108539 (+ 6041us) server_negotiation.cc:407] Connection header received
0620 02:26:17.108740 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:17.108747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:17.108920 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:17.109041 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:17.112021 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.113174 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.115271 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.115926 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.116030 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:17.117602 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:17.117614 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:17.117632 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:17.118417 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:17.118481 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:17.118511 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:17.118625 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:17.118866 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 02:26:17.118995 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:26:17.120668 28119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:17.129863 28429 ts_tablet_manager.cc:1918] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:17.130082 28429 log.cc:1196] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/39dafdc38f6c4746944736134f9929bd
I20260620 02:26:17.130627 28429 ts_tablet_manager.cc:1939] T 39dafdc38f6c4746944736134f9929bd P dbede4f033d9402ca7c9cf6c288c98dc: Deleting consensus metadata
I20260620 02:26:17.136477 28428 ts_tablet_manager.cc:1905] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:17.140662 27925 catalog_manager.cc:4841] TS dbede4f033d9402ca7c9cf6c288c98dc (127.9.231.66:42827): tablet 39dafdc38f6c4746944736134f9929bd (table test-table [id=5b956bdf27e34960bdd4ae3ad9871fee]) successfully deleted
I20260620 02:26:17.146538 28428 ts_tablet_manager.cc:1918] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:17.146800 28428 log.cc:1196] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/39dafdc38f6c4746944736134f9929bd
I20260620 02:26:17.147511 28428 ts_tablet_manager.cc:1939] T 39dafdc38f6c4746944736134f9929bd P 5263165190ff4097afd25e53c66e11f9: Deleting consensus metadata
I20260620 02:26:17.150147 27924 catalog_manager.cc:4841] TS 5263165190ff4097afd25e53c66e11f9 (127.9.231.67:46439): tablet 39dafdc38f6c4746944736134f9929bd (table test-table [id=5b956bdf27e34960bdd4ae3ad9871fee]) successfully deleted
I20260620 02:26:17.255079 28427 tablet_replica.cc:330] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6: stopping tablet replica
I20260620 02:26:17.255621 28427 raft_consensus.cc:2229] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:17.256260 28427 raft_consensus.cc:2258] T 39dafdc38f6c4746944736134f9929bd P 7825c73034464e4182195e113c15c8e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:17.256680 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27997
I20260620 02:26:17.282773 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 28128
W20260620 02:26:17.282780 27923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.231.65:43561 (error 108)
I20260620 02:26:17.285519 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.284870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57100)
0620 02:26:17.285147 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:17.285310 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260620 02:26:17.285833 27923 catalog_manager.cc:4584] TS 7825c73034464e4182195e113c15c8e6 (127.9.231.65:43561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39dafdc38f6c4746944736134f9929bd: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
I20260620 02:26:17.292663 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.283865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.9.231.66:42827)
0620 02:26:17.284494 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:17.284500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:17.284518 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:26:17.284583 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:17.284588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:17.284694 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:17.284801 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:17.285459 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.286738 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.288474 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.289235 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.289368 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:17.290824 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:17.290837 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:17.290859 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:17.291758 (+ 899us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:17.291825 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:17.291850 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:17.291967 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:17.292128 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 02:26:17.292247 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504}
I20260620 02:26:17.293430 28250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:17.339548 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.338961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57108)
0620 02:26:17.339241 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:17.339357 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 02:26:17.340744 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.340339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57120)
0620 02:26:17.340503 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:17.340588 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 02:26:17.401744 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.401133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57134)
0620 02:26:17.401389 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:17.401516 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 02:26:17.403230 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.402743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57148)
0620 02:26:17.402924 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:17.403012 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 02:26:17.412791 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 28128
I20260620 02:26:17.433058 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 28258
I20260620 02:26:17.440435 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.433894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.9.231.67:46439)
0620 02:26:17.434062 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:17.434067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:17.434085 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:26:17.434281 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:17.434285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:17.434369 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:17.434465 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:17.434939 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.435757 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.437375 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:17.437775 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:17.437869 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:17.439045 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:17.439130 (+ 85us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:17.439154 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:17.439752 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:17.439804 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:17.439825 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:17.439916 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:17.440062 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 02:26:17.440170 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:26:17.441187 28380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:17.484210 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.483565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57150)
0620 02:26:17.483835 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:17.484000 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 02:26:17.485546 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:17.485074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43561 (local address 127.0.0.1:57166)
0620 02:26:17.485314 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:17.485405 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:43561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 02:26:17.535364 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 28258
I20260620 02:26:17.555045 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 27907
I20260620 02:26:17.556548 27968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:17.670073 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 27907
2026-06-20T02:26:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3112 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:17 dist-test-slave-9vg1 krb5kdc[28438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:17 dist-test-slave-9vg1 krb5kdc[28438](info): set up 2 sockets
Jun 20 02:26:17 dist-test-slave-9vg1 krb5kdc[28438](info): commencing operation
krb5kdc: starting...
W20260620 02:26:18.816655 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:18 dist-test-slave-9vg1 krb5kdc[28438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:26:19.096545 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:42393
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:19.449671 28451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:19.450073 28451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:19.450256 28451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:19.457866 28451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:19.458062 28451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:19.458179 28451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:19.458295 28451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:19.458385 28451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:19.468919 28451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:19.470427 28451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:19.471866 28451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:19.484305 28456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:19.484230 28457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:19.484277 28459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:19.485634 28451 server_base.cc:1034] running on GCE node
I20260620 02:26:19.486572 28451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:19.488248 28451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:19.489506 28451 hybrid_clock.cc:648] HybridClock initialized: now 1781922379489447 us; error 81 us; skew 500 ppm
I20260620 02:26:19.489882 28451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:19 dist-test-slave-9vg1 krb5kdc[28438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922379, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:19.493257 28451 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:26:19.495122 28451 webserver.cc:460] Webserver started at http://127.9.231.126:34259/ using document root <none> and password file <none>
I20260620 02:26:19.495927 28451 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:19.496084 28451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:19.496417 28451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:19.499709 28451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4"
format_stamp: "Formatted at 2026-06-20 02:26:19 on dist-test-slave-9vg1"
server_key: "1d84126c9ad54a0a36be8caddc55c6ea"
server_key_iv: "0559c92861c4191fe6f494bd63c95c26"
server_key_version: "encryptionkey@0"
I20260620 02:26:19.500540 28451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4"
format_stamp: "Formatted at 2026-06-20 02:26:19 on dist-test-slave-9vg1"
server_key: "1d84126c9ad54a0a36be8caddc55c6ea"
server_key_iv: "0559c92861c4191fe6f494bd63c95c26"
server_key_version: "encryptionkey@0"
I20260620 02:26:19.506701 28451 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:26:19.510787 28466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:19.512341 28451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:19.512562 28451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4"
format_stamp: "Formatted at 2026-06-20 02:26:19 on dist-test-slave-9vg1"
server_key: "1d84126c9ad54a0a36be8caddc55c6ea"
server_key_iv: "0559c92861c4191fe6f494bd63c95c26"
server_key_version: "encryptionkey@0"
I20260620 02:26:19.512825 28451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:19.540161 28451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:19.544831 28451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:19.545161 28451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:19.565321 28451 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:42393
I20260620 02:26:19.565357 28517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:42393 every 8 connection(s)
I20260620 02:26:19.566847 28451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:19.571022 28518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:19.574034 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 28451
I20260620 02:26:19.574331 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:19.574800 10141 external_mini_cluster.cc:1428] Setting key 37ae3846b0ff60201c94a687f67fecc0
I20260620 02:26:19.583302 28518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4: Bootstrap starting.
I20260620 02:26:19.586865 28518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:19.588100 28518 log.cc:824] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:26:19 dist-test-slave-9vg1 krb5kdc[28438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:19.592217 28518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4: No bootstrap required, opened a new log
I20260620 02:26:19.600723 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:19.576947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.9.231.126:42393)
0620 02:26:19.577758 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:19.577770 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:19.577811 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:26:19.579627 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:19.579657 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:19.580079 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:19.580491 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:19.582083 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:19.583552 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:19.584948 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:19.585666 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:19.585863 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:19.591704 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:19.591713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:19.591730 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:19.591774 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:19.595454 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:19.596522 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:19.596535 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:19.596544 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:19.596771 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:19.597279 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:19.597283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:19.597285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:19.597960 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:19.598237 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:19.598849 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 02:26:19.599496 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":337,"threads_started":1}
I20260620 02:26:19.602666 28518 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42393 } }
I20260620 02:26:19.603300 28518 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:19.603444 28518 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e5b3c4aeb74461b0be2a8a31db9ed4, State: Initialized, Role: FOLLOWER
I20260620 02:26:19.604212 28518 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42393 } }
I20260620 02:26:19.604539 28518 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:19.604693 28518 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:19.604846 28518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:19.607366 28518 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42393 } }
I20260620 02:26:19.607903 28518 leader_election.cc:302] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4e5b3c4aeb74461b0be2a8a31db9ed4; no voters:
I20260620 02:26:19.608644 28518 leader_election.cc:288] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:19.608804 28523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:19.609694 28523 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [term 1 LEADER]: Becoming Leader. State: Replica: a4e5b3c4aeb74461b0be2a8a31db9ed4, State: Running, Role: LEADER
I20260620 02:26:19.610399 28523 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42393 } }
I20260620 02:26:19.611562 28518 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:19.613745 28525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4e5b3c4aeb74461b0be2a8a31db9ed4. Latest consensus state: current_term: 1 leader_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42393 } } }
I20260620 02:26:19.614308 28525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:19.615593 28524 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e5b3c4aeb74461b0be2a8a31db9ed4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 42393 } } }
I20260620 02:26:19.616096 28524 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:19.616398 28528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:19.621611 28528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:19.631863 28528 catalog_manager.cc:1347] Generated new cluster ID: 06cf234a456343c8b3e77c897bbeee16
I20260620 02:26:19.632040 28528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:19.655856 28528 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:19.657394 28528 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:19.674654 28528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4: Generated new TSK 0
I20260620 02:26:19.675843 28528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:26:19.751915 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42393
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:20.138728 28544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:20.139142 28544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:20.139335 28544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:20.147403 28544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:20.147598 28544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:20.147799 28544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:26:20.158514 28544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:20.160141 28544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:20.161688 28544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:20.177119 28552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:20.177157 28550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:20.177088 28549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:20.178236 28544 server_base.cc:1034] running on GCE node
I20260620 02:26:20.179008 28544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:20.180230 28544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:20.181519 28544 hybrid_clock.cc:648] HybridClock initialized: now 1781922380181451 us; error 68 us; skew 500 ppm
I20260620 02:26:20.181936 28544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:20 dist-test-slave-9vg1 krb5kdc[28438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:20.185622 28544 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:26:20.187523 28544 webserver.cc:460] Webserver started at http://127.9.231.65:44553/ using document root <none> and password file <none>
I20260620 02:26:20.188279 28544 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:20.188438 28544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:20.188778 28544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:20.192011 28544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "f1e0a0061a844664b775777ed7be0a10"
format_stamp: "Formatted at 2026-06-20 02:26:20 on dist-test-slave-9vg1"
server_key: "82188d4eeef004b1989b78019693a282"
server_key_iv: "157a99e77c7bcc4b583ca3097cd5d7d7"
server_key_version: "encryptionkey@0"
I20260620 02:26:20.192834 28544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "f1e0a0061a844664b775777ed7be0a10"
format_stamp: "Formatted at 2026-06-20 02:26:20 on dist-test-slave-9vg1"
server_key: "82188d4eeef004b1989b78019693a282"
server_key_iv: "157a99e77c7bcc4b583ca3097cd5d7d7"
server_key_version: "encryptionkey@0"
I20260620 02:26:20.198637 28544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:26:20.202808 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:20.204384 28544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:20.204613 28544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "f1e0a0061a844664b775777ed7be0a10"
format_stamp: "Formatted at 2026-06-20 02:26:20 on dist-test-slave-9vg1"
server_key: "82188d4eeef004b1989b78019693a282"
server_key_iv: "157a99e77c7bcc4b583ca3097cd5d7d7"
server_key_version: "encryptionkey@0"
I20260620 02:26:20.204857 28544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:20.223953 28544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:20.228416 28544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:20.228778 28544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:20.229931 28544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:20.231491 28544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:20.231604 28544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:20.231771 28544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:20.231846 28544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:20.267750 28544 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:43187
I20260620 02:26:20.267848 28671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:43187 every 8 connection(s)
I20260620 02:26:20.269161 28544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:26:20.274085 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 28544
I20260620 02:26:20.274325 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:26:20.274920 10141 external_mini_cluster.cc:1428] Setting key a832a764c4da2e9bb2b1522bbcb988a8
Jun 20 02:26:20 dist-test-slave-9vg1 krb5kdc[28438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:20.294661 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:20.271977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:45447 (local address 127.9.231.126:42393)
0620 02:26:20.272479 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:20.272484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:20.273461 (+ 977us) server_negotiation.cc:407] Connection header received
0620 02:26:20.276322 (+ 2861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:20.276327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:20.276416 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:20.276554 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:20.278879 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.279939 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.282643 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.283487 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.283646 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:20.288104 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:20.288111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:20.288114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:20.288166 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:20.291161 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:20.292050 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:20.292057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:20.292061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:20.292206 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:20.292715 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:20.292721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:20.292724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:20.293027 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:20.293189 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:20.294004 (+ 815us) server_negotiation.cc:299] Negotiation successful
0620 02:26:20.294261 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":284,"threads_started":1}
I20260620 02:26:20.295845 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:20.272450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:42393 (local address 127.9.231.65:45447)
0620 02:26:20.273272 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:20.273319 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:20.275955 (+ 2636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:20.276893 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:20.276910 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:20.277747 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:20.278660 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:20.278691 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.280117 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.280125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:20.282407 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:20.282429 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.283776 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.283784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:20.284051 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:20.284810 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:20.284841 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:20.287809 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:20.291330 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:20.291341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:20.291361 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:20.291884 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:20.292337 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:20.292343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:20.292346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:20.292589 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:20.293176 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:20.293184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:20.293603 (+ 419us) client_negotiation.cc:769] Sending connection context
0620 02:26:20.293896 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:26:20.294499 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":290,"threads_started":1}
I20260620 02:26:20.297780 28672 heartbeater.cc:347] Connected to a master server at 127.9.231.126:42393
I20260620 02:26:20.298125 28672 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:20.298947 28672 heartbeater.cc:511] Master 127.9.231.126:42393 requested a full tablet report, sending...
I20260620 02:26:20.302055 28483 ts_manager.cc:194] Registered new tserver with Master: f1e0a0061a844664b775777ed7be0a10 (127.9.231.65:43187)
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
I20260620 02:26:20.303973 28483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:45447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:26:20.340809 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42393
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:20.700614 28678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:20.700927 28678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:20.701022 28678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:20.708988 28678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:20.709124 28678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:20.709245 28678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:26:20.720211 28678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:20.721526 28678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:20.722858 28678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:20.737255 28684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:20.737826 28686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:20.737435 28683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:20.738623 28678 server_base.cc:1034] running on GCE node
I20260620 02:26:20.739428 28678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:20.740513 28678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:20.741753 28678 hybrid_clock.cc:648] HybridClock initialized: now 1781922380741702 us; error 42 us; skew 500 ppm
I20260620 02:26:20.742153 28678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:20 dist-test-slave-9vg1 krb5kdc[28438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:20.745857 28678 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:26:20.747670 28678 webserver.cc:460] Webserver started at http://127.9.231.66:36175/ using document root <none> and password file <none>
I20260620 02:26:20.748378 28678 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:20.748484 28678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:20.748771 28678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:20.752213 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "70f09675682c4f2695b681adef357e58"
format_stamp: "Formatted at 2026-06-20 02:26:20 on dist-test-slave-9vg1"
server_key: "28cf43f21a80681c4d488323adb11275"
server_key_iv: "6058cc0d14a956ce4bbfeb8f75e71fde"
server_key_version: "encryptionkey@0"
I20260620 02:26:20.753039 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "70f09675682c4f2695b681adef357e58"
format_stamp: "Formatted at 2026-06-20 02:26:20 on dist-test-slave-9vg1"
server_key: "28cf43f21a80681c4d488323adb11275"
server_key_iv: "6058cc0d14a956ce4bbfeb8f75e71fde"
server_key_version: "encryptionkey@0"
I20260620 02:26:20.758708 28678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:26:20.762508 28693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:20.764178 28678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:20.764385 28678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "70f09675682c4f2695b681adef357e58"
format_stamp: "Formatted at 2026-06-20 02:26:20 on dist-test-slave-9vg1"
server_key: "28cf43f21a80681c4d488323adb11275"
server_key_iv: "6058cc0d14a956ce4bbfeb8f75e71fde"
server_key_version: "encryptionkey@0"
I20260620 02:26:20.764626 28678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:20.783142 28678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:20.788244 28678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:20.788653 28678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:20.789742 28678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:20.791322 28678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:20.791443 28678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:20.791611 28678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:20.791684 28678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:20.828402 28678 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:38751
I20260620 02:26:20.828507 28805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:38751 every 8 connection(s)
I20260620 02:26:20.829805 28678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:26:20.838946 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 28678
I20260620 02:26:20.839200 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:26:20.839677 10141 external_mini_cluster.cc:1428] Setting key 02e569d830aa42366762a909879b385f
Jun 20 02:26:20 dist-test-slave-9vg1 krb5kdc[28438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:20.853837 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:20.832799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:55845 (local address 127.9.231.126:42393)
0620 02:26:20.833319 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:20.833326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:20.834400 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 02:26:20.836477 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:20.836482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:20.836559 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:20.836675 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:20.838358 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.839214 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.841175 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.841918 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.842072 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:20.846375 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:20.846381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:20.846385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:20.846425 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:20.849688 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:20.850873 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:20.850882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:20.850888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:20.851129 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:20.851630 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:20.851637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:20.851642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:20.852088 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:20.852261 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:20.853019 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 02:26:20.853322 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260620 02:26:20.854580 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:20.833433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:42393 (local address 127.9.231.66:55845)
0620 02:26:20.834219 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:20.834259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:20.836155 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:20.836924 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:20.836933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:20.837510 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:20.838157 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:20.838178 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.839443 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.839447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:20.841007 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:20.841021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:20.842195 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:20.842203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:20.842461 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:20.843283 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:20.843308 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:20.846043 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:20.849913 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:20.849928 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:20.849952 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:20.850692 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:20.851286 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:20.851290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:20.851292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:20.851504 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:20.852236 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:20.852243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:20.852620 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 02:26:20.852872 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 02:26:20.853334 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":322,"threads_started":1}
I20260620 02:26:20.856360 28806 heartbeater.cc:347] Connected to a master server at 127.9.231.126:42393
I20260620 02:26:20.856715 28806 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:20.857712 28806 heartbeater.cc:511] Master 127.9.231.126:42393 requested a full tablet report, sending...
I20260620 02:26:20.859822 28483 ts_manager.cc:194] Registered new tserver with Master: 70f09675682c4f2695b681adef357e58 (127.9.231.66:38751)
I20260620 02:26:20.861243 28483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:55845
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:26:20.904915 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42393
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:21.260553 28812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:21.260932 28812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:21.261080 28812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:21.268815 28812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:21.268978 28812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:21.269188 28812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:26:21.279554 28812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:42393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:21.280997 28812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:21.282478 28812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:21.297144 28817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:21.297148 28818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:21.297107 28820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:21.298511 28812 server_base.cc:1034] running on GCE node
I20260620 02:26:21.299401 28812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:21.300573 28812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:21.301864 28812 hybrid_clock.cc:648] HybridClock initialized: now 1781922381301769 us; error 79 us; skew 500 ppm
I20260620 02:26:21.302287 28812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:21 dist-test-slave-9vg1 krb5kdc[28438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922381, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:21.306811 28812 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:26:21.307706 28672 heartbeater.cc:503] Master 127.9.231.126:42393 was elected leader, sending a full tablet report...
I20260620 02:26:21.310074 28812 webserver.cc:460] Webserver started at http://127.9.231.67:43295/ using document root <none> and password file <none>
I20260620 02:26:21.310876 28812 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:21.311030 28812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:21.311455 28812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:21.314683 28812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "35ac8d1f34bd4ed48dcef3a480ac681b"
format_stamp: "Formatted at 2026-06-20 02:26:21 on dist-test-slave-9vg1"
server_key: "c645c890c88c7f4e6881d5ac3b88deaa"
server_key_iv: "1fca72df695b44948ed22c1ba3f201c5"
server_key_version: "encryptionkey@0"
I20260620 02:26:21.315542 28812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "35ac8d1f34bd4ed48dcef3a480ac681b"
format_stamp: "Formatted at 2026-06-20 02:26:21 on dist-test-slave-9vg1"
server_key: "c645c890c88c7f4e6881d5ac3b88deaa"
server_key_iv: "1fca72df695b44948ed22c1ba3f201c5"
server_key_version: "encryptionkey@0"
I20260620 02:26:21.321349 28812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:26:21.325013 28827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:21.326606 28812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:26:21.326826 28812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "35ac8d1f34bd4ed48dcef3a480ac681b"
format_stamp: "Formatted at 2026-06-20 02:26:21 on dist-test-slave-9vg1"
server_key: "c645c890c88c7f4e6881d5ac3b88deaa"
server_key_iv: "1fca72df695b44948ed22c1ba3f201c5"
server_key_version: "encryptionkey@0"
I20260620 02:26:21.327069 28812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:21.344430 28812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:21.349442 28812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:21.349778 28812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:21.350872 28812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:21.352509 28812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:21.352622 28812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:21.352782 28812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:21.352871 28812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:21.389842 28812 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:44285
I20260620 02:26:21.389875 28939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:44285 every 8 connection(s)
I20260620 02:26:21.391305 28812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:26:21.393833 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 28812
I20260620 02:26:21.394024 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:26:21.394732 10141 external_mini_cluster.cc:1428] Setting key ec6fe2bae2a6556442abff8611a2f480
Jun 20 02:26:21 dist-test-slave-9vg1 krb5kdc[28438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922381, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:21.415513 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.394400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:56951 (local address 127.9.231.126:42393)
0620 02:26:21.394946 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.394952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.396092 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 02:26:21.398601 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.398607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.398718 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.398842 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:21.400581 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.401381 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.403048 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.403516 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.403680 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.407633 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:21.407640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:21.407643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:21.407706 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:21.411767 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.412719 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.412726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.412730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.412871 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.413359 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.413362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.413364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.413734 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:21.413863 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.414869 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.415159 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":255,"threads_started":1}
I20260620 02:26:21.416505 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.394986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:42393 (local address 127.9.231.67:56951)
0620 02:26:21.395916 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:21.395970 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:21.398322 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:21.399181 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:21.399190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:21.399762 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:21.400393 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.400412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.401559 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.401565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.402877 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.402890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.403788 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.403798 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.404058 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.404776 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:21.404798 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:21.407346 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:21.411949 (+ 4603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:21.411958 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:21.411985 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:21.412532 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:21.413008 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:21.413011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:21.413014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:21.413232 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:21.413864 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:21.413870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:21.414262 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 02:26:21.414522 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 02:26:21.414962 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":343,"threads_started":1}
I20260620 02:26:21.418462 28940 heartbeater.cc:347] Connected to a master server at 127.9.231.126:42393
I20260620 02:26:21.418876 28940 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:21.419780 28940 heartbeater.cc:511] Master 127.9.231.126:42393 requested a full tablet report, sending...
I20260620 02:26:21.421552 28483 ts_manager.cc:194] Registered new tserver with Master: 35ac8d1f34bd4ed48dcef3a480ac681b (127.9.231.67:44285)
I20260620 02:26:21.422720 28483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:56951
I20260620 02:26:21.424692 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:26:21.445148 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.429091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.9.231.126:42393)
0620 02:26:21.429300 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.429305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.429682 (+ 377us) server_negotiation.cc:407] Connection header received
0620 02:26:21.429842 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.429846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.429919 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.430035 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:21.432311 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.433283 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.434905 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.435596 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.435724 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.438686 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:21.438691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:21.438695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:21.438733 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:21.441672 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.442845 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.442850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.442852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.443129 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.443696 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.443699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.443701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.444028 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:21.444235 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.444755 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.444910 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:26:21.450145 28483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:26:21.457839 28483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:26:21.485029 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.472131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:44285 (local address 127.0.0.1:46038)
0620 02:26:21.473110 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:21.473168 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:21.473345 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:21.475894 (+ 2549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:21.475898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:21.475926 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:21.476452 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.476466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.478441 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.478447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.481581 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.481598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.483836 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.483842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.484022 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.484083 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 02:26:21.484260 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 02:26:21.484506 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":123,"thread_start_us":531,"threads_started":1}
I20260620 02:26:21.485527 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.472625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43187 (local address 127.0.0.1:50116)
0620 02:26:21.474529 (+ 1904us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:21.474543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:21.474690 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:21.475485 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:21.475498 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:21.475557 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:21.476307 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.476332 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.478782 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.478787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.481408 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.481424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.483610 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.483617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.484709 (+ 1092us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.484746 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:26:21.484844 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:26:21.485018 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1796,"mutex_wait_us":827,"thread_start_us":1645,"threads_started":1}
I20260620 02:26:21.486387 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.472811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.9.231.67:44285)
0620 02:26:21.475347 (+ 2536us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.475357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.475389 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:26:21.475487 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.475495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.475714 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.475907 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:21.476690 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.478284 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.481769 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.483347 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.483518 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.483801 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.484333 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.484532 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2367,"thread_start_us":2172,"threads_started":1}
I20260620 02:26:21.486533 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.473284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.9.231.65:43187)
0620 02:26:21.474086 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.474098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.474663 (+ 565us) server_negotiation.cc:407] Connection header received
0620 02:26:21.474873 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.474884 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.475200 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.475408 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:21.476608 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.478555 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.481642 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.483347 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.483521 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.483801 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.485918 (+ 2117us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.486127 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":461,"threads_started":1}
I20260620 02:26:21.488384 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.473640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:38751 (local address 127.0.0.1:35158)
0620 02:26:21.478410 (+ 4770us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:21.478430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:21.478554 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:21.479417 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:21.479423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:21.479472 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:21.480100 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.480125 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.482245 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.482261 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.484351 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.484366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.486308 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.486316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.487802 (+ 1486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.487852 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 02:26:21.487962 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:26:21.488076 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4626,"mutex_wait_us":1,"thread_start_us":4448,"threads_started":1}
I20260620 02:26:21.488543 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.474255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.9.231.66:38751)
0620 02:26:21.477941 (+ 3686us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.477950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.478509 (+ 559us) server_negotiation.cc:407] Connection header received
0620 02:26:21.478774 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.478782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.479050 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.479303 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:21.480350 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.481999 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.484535 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.486066 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.486225 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.486442 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.488033 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.488257 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3523,"thread_start_us":3400,"threads_started":1}
I20260620 02:26:21.490903 28875 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4ac34708534d72b863c27d4abeb6b6 (DEFAULT_TABLE table=test-table [id=192bf1759a6947ffbbc9926508daa093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:21.492623 28875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4ac34708534d72b863c27d4abeb6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:21.492228 28741 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4ac34708534d72b863c27d4abeb6b6 (DEFAULT_TABLE table=test-table [id=192bf1759a6947ffbbc9926508daa093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:21.492247 28607 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4ac34708534d72b863c27d4abeb6b6 (DEFAULT_TABLE table=test-table [id=192bf1759a6947ffbbc9926508daa093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:21.493515 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4ac34708534d72b863c27d4abeb6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:21.493566 28741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4ac34708534d72b863c27d4abeb6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:21.507272 28959 tablet_bootstrap.cc:492] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Bootstrap starting.
I20260620 02:26:21.509819 28961 tablet_bootstrap.cc:492] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Bootstrap starting.
I20260620 02:26:21.511791 28959 tablet_bootstrap.cc:654] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:21.511854 28960 tablet_bootstrap.cc:492] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Bootstrap starting.
I20260620 02:26:21.513545 28959 log.cc:824] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:21.514468 28961 tablet_bootstrap.cc:654] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:21.515384 28960 tablet_bootstrap.cc:654] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:21.516253 28961 log.cc:824] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:21.516525 28960 log.cc:824] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:21.520278 28961 tablet_bootstrap.cc:492] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: No bootstrap required, opened a new log
I20260620 02:26:21.520316 28960 tablet_bootstrap.cc:492] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: No bootstrap required, opened a new log
I20260620 02:26:21.520613 28961 ts_tablet_manager.cc:1397] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 02:26:21.520617 28960 ts_tablet_manager.cc:1397] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 02:26:21.520663 28959 tablet_bootstrap.cc:492] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: No bootstrap required, opened a new log
I20260620 02:26:21.520902 28959 ts_tablet_manager.cc:1397] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260620 02:26:21.529649 28961 raft_consensus.cc:348] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.530107 28961 raft_consensus.cc:374] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:21.530238 28961 raft_consensus.cc:729] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ac8d1f34bd4ed48dcef3a480ac681b, State: Initialized, Role: FOLLOWER
I20260620 02:26:21.530949 28961 consensus_queue.cc:260] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.531903 28960 raft_consensus.cc:348] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.532521 28960 raft_consensus.cc:374] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:21.532713 28960 raft_consensus.cc:729] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f09675682c4f2695b681adef357e58, State: Initialized, Role: FOLLOWER
I20260620 02:26:21.532796 28940 heartbeater.cc:503] Master 127.9.231.126:42393 was elected leader, sending a full tablet report...
I20260620 02:26:21.532917 28959 raft_consensus.cc:348] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.533376 28959 raft_consensus.cc:374] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:21.533484 28959 raft_consensus.cc:729] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e0a0061a844664b775777ed7be0a10, State: Initialized, Role: FOLLOWER
I20260620 02:26:21.533402 28960 consensus_queue.cc:260] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.534235 28959 consensus_queue.cc:260] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.534598 28806 heartbeater.cc:503] Master 127.9.231.126:42393 was elected leader, sending a full tablet report...
I20260620 02:26:21.535506 28960 ts_tablet_manager.cc:1428] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260620 02:26:21.537446 28959 ts_tablet_manager.cc:1428] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260620 02:26:21.537606 28961 ts_tablet_manager.cc:1428] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260620 02:26:21.566883 28966 raft_consensus.cc:482] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:26:21.567337 28966 raft_consensus.cc:504] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.569312 28966 leader_election.cc:288] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ac8d1f34bd4ed48dcef3a480ac681b (127.9.231.67:44285), f1e0a0061a844664b775777ed7be0a10 (127.9.231.65:43187)
I20260620 02:26:21.578312 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.569886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:57131 (local address 127.9.231.65:43187)
0620 02:26:21.570141 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.570147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.571496 (+ 1349us) server_negotiation.cc:407] Connection header received
0620 02:26:21.571662 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.571678 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.571755 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.571858 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:21.572580 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.573654 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.575928 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.577193 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.577393 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.577475 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.577841 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.578006 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:26:21.578578 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.570417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:44511 (local address 127.9.231.67:44285)
0620 02:26:21.570595 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.570600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.570823 (+ 223us) server_negotiation.cc:407] Connection header received
0620 02:26:21.571274 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.571282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.571377 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.571456 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:21.572308 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.573501 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.576303 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.577574 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.577756 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.577836 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.578198 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.578339 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:26:21.578732 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.569761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:44285 (local address 127.9.231.66:44511)
0620 02:26:21.570666 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:21.570693 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:21.571017 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:21.571702 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:21.571707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:21.571732 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:21.572149 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.572161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.573664 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.573670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.576096 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.576114 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.577835 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.577842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.578021 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.578052 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:26:21.578144 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:26:21.578300 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":88,"thread_start_us":971,"threads_started":1}
I20260620 02:26:21.579250 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.569783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:43187 (local address 127.9.231.66:57131)
0620 02:26:21.571363 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:21.571380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:21.571514 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:21.571902 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:21.571907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:21.571941 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:21.572395 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.572407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.573808 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.573813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.575713 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:21.575732 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.577420 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.577428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:21.577600 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.577627 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:26:21.578687 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0620 02:26:21.578849 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"mutex_wait_us":43,"thread_start_us":1268,"threads_started":1}
I20260620 02:26:21.580080 28895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4ac34708534d72b863c27d4abeb6b6" candidate_uuid: "70f09675682c4f2695b681adef357e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" is_pre_election: true
I20260620 02:26:21.580372 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4ac34708534d72b863c27d4abeb6b6" candidate_uuid: "70f09675682c4f2695b681adef357e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e0a0061a844664b775777ed7be0a10" is_pre_election: true
I20260620 02:26:21.580672 28895 raft_consensus.cc:2454] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70f09675682c4f2695b681adef357e58 in term 0.
I20260620 02:26:21.580942 28627 raft_consensus.cc:2454] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70f09675682c4f2695b681adef357e58 in term 0.
I20260620 02:26:21.581545 28697 leader_election.cc:302] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ac8d1f34bd4ed48dcef3a480ac681b, 70f09675682c4f2695b681adef357e58; no voters:
I20260620 02:26:21.582050 28966 raft_consensus.cc:2781] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260620 02:26:21.582062 28807 tablet.cc:2369] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:21.582273 28966 raft_consensus.cc:482] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:26:21.582391 28966 raft_consensus.cc:3037] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:21.584832 28966 raft_consensus.cc:504] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.585577 28966 leader_election.cc:288] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [CANDIDATE]: Term 1 election: Requested vote from peers 35ac8d1f34bd4ed48dcef3a480ac681b (127.9.231.67:44285), f1e0a0061a844664b775777ed7be0a10 (127.9.231.65:43187)
I20260620 02:26:21.586186 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4ac34708534d72b863c27d4abeb6b6" candidate_uuid: "70f09675682c4f2695b681adef357e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e0a0061a844664b775777ed7be0a10"
I20260620 02:26:21.586253 28895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4ac34708534d72b863c27d4abeb6b6" candidate_uuid: "70f09675682c4f2695b681adef357e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b"
I20260620 02:26:21.586414 28627 raft_consensus.cc:3037] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:21.586513 28895 raft_consensus.cc:3037] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:21.589057 28895 raft_consensus.cc:2454] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70f09675682c4f2695b681adef357e58 in term 1.
I20260620 02:26:21.589098 28627 raft_consensus.cc:2454] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70f09675682c4f2695b681adef357e58 in term 1.
I20260620 02:26:21.589656 28696 leader_election.cc:302] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35ac8d1f34bd4ed48dcef3a480ac681b, 70f09675682c4f2695b681adef357e58, f1e0a0061a844664b775777ed7be0a10; no voters:
I20260620 02:26:21.590080 28966 raft_consensus.cc:2781] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:21.590662 28966 raft_consensus.cc:686] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 1 LEADER]: Becoming Leader. State: Replica: 70f09675682c4f2695b681adef357e58, State: Running, Role: LEADER
I20260620 02:26:21.591410 28966 consensus_queue.cc:237] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } }
I20260620 02:26:21.597424 28483 catalog_manager.cc:5495] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70f09675682c4f2695b681adef357e58 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "70f09675682c4f2695b681adef357e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70f09675682c4f2695b681adef357e58" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 } health_report { overall_health: UNKNOWN } } }
W20260620 02:26:21.643307 28941 tablet.cc:2369] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:21.646627 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.639800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.9.231.66:38751)
0620 02:26:21.640063 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.640070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.640173 (+ 103us) server_negotiation.cc:407] Connection header received
0620 02:26:21.640357 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.640361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.640444 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.640647 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:26:21.641109 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.641913 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.643928 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.644401 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.644513 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.645159 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:26:21.645744 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:26:21.645916 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.646161 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.646255 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:26:21.656174 28895 raft_consensus.cc:1261] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 1 FOLLOWER]: Refusing update from remote peer 70f09675682c4f2695b681adef357e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:21.656180 28627 raft_consensus.cc:1261] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 1 FOLLOWER]: Refusing update from remote peer 70f09675682c4f2695b681adef357e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:21.657507 28970 consensus_queue.cc:1035] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ac8d1f34bd4ed48dcef3a480ac681b" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:21.657895 28966 consensus_queue.cc:1035] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e0a0061a844664b775777ed7be0a10" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:21.685323 28483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:21.685578 28483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:21.689337 28483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4e5b3c4aeb74461b0be2a8a31db9ed4: Sending DeleteTablet for 3 replicas of tablet 0a4ac34708534d72b863c27d4abeb6b6
I20260620 02:26:21.690748 28875 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4ac34708534d72b863c27d4abeb6b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:21 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:46038
I20260620 02:26:21.690755 28741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4ac34708534d72b863c27d4abeb6b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:21 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:35158
I20260620 02:26:21.690816 28607 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4ac34708534d72b863c27d4abeb6b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:21 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:50116
I20260620 02:26:21.696132 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 28544
I20260620 02:26:21.704229 28983 tablet_replica.cc:330] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: stopping tablet replica
I20260620 02:26:21.704456 28984 tablet_replica.cc:330] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: stopping tablet replica
I20260620 02:26:21.704857 28983 raft_consensus.cc:2229] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:21.705125 28984 raft_consensus.cc:2229] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:21.705534 28983 raft_consensus.cc:2258] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:21.705739 28984 raft_consensus.cc:2258] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:21.708170 28984 ts_tablet_manager.cc:1905] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:21.708253 28983 ts_tablet_manager.cc:1905] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:21.716248 28984 ts_tablet_manager.cc:1918] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:21.716660 28984 log.cc:1196] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/0a4ac34708534d72b863c27d4abeb6b6
I20260620 02:26:21.717345 28984 ts_tablet_manager.cc:1939] T 0a4ac34708534d72b863c27d4abeb6b6 P 35ac8d1f34bd4ed48dcef3a480ac681b: Deleting consensus metadata
I20260620 02:26:21.718170 28983 ts_tablet_manager.cc:1918] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:21.718428 28983 log.cc:1196] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/0a4ac34708534d72b863c27d4abeb6b6
I20260620 02:26:21.719152 28983 ts_tablet_manager.cc:1939] T 0a4ac34708534d72b863c27d4abeb6b6 P f1e0a0061a844664b775777ed7be0a10: Deleting consensus metadata
I20260620 02:26:21.719995 28470 catalog_manager.cc:4841] TS 35ac8d1f34bd4ed48dcef3a480ac681b (127.9.231.67:44285): tablet 0a4ac34708534d72b863c27d4abeb6b6 (table test-table [id=192bf1759a6947ffbbc9926508daa093]) successfully deleted
I20260620 02:26:21.721284 28469 catalog_manager.cc:4841] TS f1e0a0061a844664b775777ed7be0a10 (127.9.231.65:43187): tablet 0a4ac34708534d72b863c27d4abeb6b6 (table test-table [id=192bf1759a6947ffbbc9926508daa093]) successfully deleted
I20260620 02:26:21.721812 28982 tablet_replica.cc:330] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: stopping tablet replica
I20260620 02:26:21.722337 28982 raft_consensus.cc:2229] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:26:21.723117 28982 raft_consensus.cc:2258] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:21.725546 28982 ts_tablet_manager.cc:1905] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:26:21 dist-test-slave-9vg1 krb5kdc[28438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:26:21.734295 28982 ts_tablet_manager.cc:1918] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:21.734519 28982 log.cc:1196] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/0a4ac34708534d72b863c27d4abeb6b6
I20260620 02:26:21.735147 28982 ts_tablet_manager.cc:1939] T 0a4ac34708534d72b863c27d4abeb6b6 P 70f09675682c4f2695b681adef357e58: Deleting consensus metadata
I20260620 02:26:21.737375 28470 catalog_manager.cc:4841] TS 70f09675682c4f2695b681adef357e58 (127.9.231.66:38751): tablet 0a4ac34708534d72b863c27d4abeb6b6 (table test-table [id=192bf1759a6947ffbbc9926508daa093]) successfully deleted
I20260620 02:26:21.738787 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.698358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.9.231.65:43187)
0620 02:26:21.698623 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.698632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.721010 (+ 22378us) server_negotiation.cc:407] Connection header received
0620 02:26:21.721236 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.721243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.721347 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.721448 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:21.723115 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.724138 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.725551 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.726054 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.726197 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.730843 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:21.730860 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:21.730877 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:21.730939 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:21.735100 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.736099 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.736118 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.736121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.736284 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.736736 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.736741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.736744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.737193 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:21.737466 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.738097 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.738359 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:26:21.739989 28667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:21.836160 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 28544
I20260620 02:26:21.856540 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 28678
Jun 20 02:26:21 dist-test-slave-9vg1 krb5kdc[28438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:26:21.875993 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:21.857513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.9.231.66:38751)
0620 02:26:21.857739 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:21.857745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:21.857876 (+ 131us) server_negotiation.cc:407] Connection header received
0620 02:26:21.858179 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:21.858186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:21.858267 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:21.858394 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:21.859839 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.860773 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.862130 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:21.862618 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:21.862781 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:21.868190 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:21.868207 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:21.868224 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:21.868273 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:21.872346 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.873409 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.873417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.873419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.873668 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:21.874160 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:21.874163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:21.874165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:21.874613 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:21.874893 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:21.875366 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 02:26:21.875587 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:26:21.876845 28801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:21.997516 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 28678
I20260620 02:26:22.018903 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 28812
Jun 20 02:26:22 dist-test-slave-9vg1 krb5kdc[28438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:26:22.037335 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:22.019870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.9.231.67:44285)
0620 02:26:22.020046 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:22.020051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:22.020136 (+ 85us) server_negotiation.cc:407] Connection header received
0620 02:26:22.020272 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:22.020276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:22.020345 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:22.020446 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:22.021992 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:22.022770 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:22.024433 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:22.024919 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:22.025067 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:22.030358 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:22.030378 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:22.030397 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:22.030445 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:22.033969 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:22.034909 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:22.034918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:22.034920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:22.035275 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:22.035707 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:22.035711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:22.035714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:22.036114 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:22.036393 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:22.036817 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 02:26:22.037026 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:26:22.038022 28935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:22.133292 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 28812
I20260620 02:26:22.151916 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 28451
I20260620 02:26:22.153354 28513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:22.271014 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 28451
2026-06-20T02:26:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4603 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T02:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:22Z Disabled control of system clock
I20260620 02:26:22.357059 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:39627
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:39627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:26:22.709899 28993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:22.710260 28993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:22.710421 28993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:22.718523 28993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:22.718657 28993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:22.718765 28993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:22.718858 28993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:22.718956 28993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:22.728984 28993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:39627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:22.730423 28993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:22.732379 28993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:22.746531 28999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:22.746747 28998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:22.746704 29001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:22.748152 28993 server_base.cc:1034] running on GCE node
I20260620 02:26:22.749207 28993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:22.750819 28993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:22.752128 28993 hybrid_clock.cc:648] HybridClock initialized: now 1781922382752069 us; error 54 us; skew 500 ppm
I20260620 02:26:22.752558 28993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:22.755288 28993 webserver.cc:460] Webserver started at http://127.9.231.126:37205/ using document root <none> and password file <none>
I20260620 02:26:22.756047 28993 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:22.756140 28993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:22.756438 28993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:22.759712 28993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "dad538b5cca94e309ddfd3e548d1c4f4"
format_stamp: "Formatted at 2026-06-20 02:26:22 on dist-test-slave-9vg1"
server_key: "fd19aaafd52dfa13c15acf767fc09534"
server_key_iv: "71657916be660cc5d07c3f5e7aec47e6"
server_key_version: "encryptionkey@0"
I20260620 02:26:22.760545 28993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "dad538b5cca94e309ddfd3e548d1c4f4"
format_stamp: "Formatted at 2026-06-20 02:26:22 on dist-test-slave-9vg1"
server_key: "fd19aaafd52dfa13c15acf767fc09534"
server_key_iv: "71657916be660cc5d07c3f5e7aec47e6"
server_key_version: "encryptionkey@0"
I20260620 02:26:22.766248 28993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 02:26:22.770321 29007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:22.771975 28993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:22.772159 28993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "dad538b5cca94e309ddfd3e548d1c4f4"
format_stamp: "Formatted at 2026-06-20 02:26:22 on dist-test-slave-9vg1"
server_key: "fd19aaafd52dfa13c15acf767fc09534"
server_key_iv: "71657916be660cc5d07c3f5e7aec47e6"
server_key_version: "encryptionkey@0"
I20260620 02:26:22.772368 28993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:22.786885 28993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:22.787868 28993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:22.788126 28993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:22.808444 28993 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:39627
I20260620 02:26:22.808450 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:39627 every 8 connection(s)
I20260620 02:26:22.809996 28993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:22.814471 29059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:22.815127 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 28993
I20260620 02:26:22.815321 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:22.815830 10141 external_mini_cluster.cc:1428] Setting key d7338085ff07d039eb70e55c55eabf1e
I20260620 02:26:22.828186 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4: Bootstrap starting.
I20260620 02:26:22.830713 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:22.817856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.9.231.126:39627)
0620 02:26:22.818995 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:22.819007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:22.819111 (+ 104us) server_negotiation.cc:407] Connection header received
0620 02:26:22.821464 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:22.821492 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:22.822068 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:22.822515 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:22.823156 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:22.824424 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:22.825864 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:22.826599 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:22.826824 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:22.828181 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:22.828191 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:22.828207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:22.828831 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:22.828918 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:22.828941 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:22.829079 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:22.829363 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 02:26:22.829714 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":435,"threads_started":1}
I20260620 02:26:22.831797 29059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:22.832971 29059 log.cc:824] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:22.836062 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4: No bootstrap required, opened a new log
I20260620 02:26:22.844453 29059 raft_consensus.cc:348] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39627 } }
I20260620 02:26:22.845012 29059 raft_consensus.cc:374] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:22.845162 29059 raft_consensus.cc:729] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad538b5cca94e309ddfd3e548d1c4f4, State: Initialized, Role: FOLLOWER
I20260620 02:26:22.845885 29059 consensus_queue.cc:260] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39627 } }
I20260620 02:26:22.846182 29059 raft_consensus.cc:388] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:22.846359 29059 raft_consensus.cc:482] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:22.846587 29059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:22.849442 29059 raft_consensus.cc:504] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39627 } }
I20260620 02:26:22.850042 29059 leader_election.cc:302] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dad538b5cca94e309ddfd3e548d1c4f4; no voters:
I20260620 02:26:22.850703 29059 leader_election.cc:288] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:22.850862 29064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:22.851720 29064 raft_consensus.cc:686] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [term 1 LEADER]: Becoming Leader. State: Replica: dad538b5cca94e309ddfd3e548d1c4f4, State: Running, Role: LEADER
I20260620 02:26:22.852308 29064 consensus_queue.cc:237] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39627 } }
I20260620 02:26:22.853106 29059 sys_catalog.cc:564] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:22.855412 29066 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dad538b5cca94e309ddfd3e548d1c4f4. Latest consensus state: current_term: 1 leader_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39627 } } }
I20260620 02:26:22.855892 29066 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:22.855414 29065 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad538b5cca94e309ddfd3e548d1c4f4" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 39627 } } }
I20260620 02:26:22.859892 29065 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:22.865108 29069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:22.871658 29069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:22.884416 29069 catalog_manager.cc:1347] Generated new cluster ID: 3d80cf4c5dd14c208d5bd9a6102440be
I20260620 02:26:22.884562 29069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:22.901453 29069 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:22.903797 29069 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:22.920809 29069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4: Generated new TSK 0
I20260620 02:26:22.922201 29069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:22.940389 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39627
--builtin_ntp_servers=127.9.231.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:26:23.317094 29083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:23.317541 29083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:23.317731 29083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:23.325428 29083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:23.325593 29083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:23.325778 29083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:26:23.335588 29083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:23.337037 29083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:23.339021 29083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:23.353997 29091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:23.354123 29089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:23.354121 29088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:23.355346 29083 server_base.cc:1034] running on GCE node
I20260620 02:26:23.356045 29083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:23.357115 29083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:23.358369 29083 hybrid_clock.cc:648] HybridClock initialized: now 1781922383358308 us; error 47 us; skew 500 ppm
I20260620 02:26:23.358707 29083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:23.361565 29083 webserver.cc:460] Webserver started at http://127.9.231.65:35077/ using document root <none> and password file <none>
I20260620 02:26:23.362296 29083 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:23.362394 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:23.362681 29083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:23.366338 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "29c8ee8609bb45b4aef795e00ee504a5"
format_stamp: "Formatted at 2026-06-20 02:26:23 on dist-test-slave-9vg1"
server_key: "bcc9c1868cde71c1b7399da0d54afd75"
server_key_iv: "dba0ace1487e80cd294ae05f519b9cc0"
server_key_version: "encryptionkey@0"
I20260620 02:26:23.367448 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "29c8ee8609bb45b4aef795e00ee504a5"
format_stamp: "Formatted at 2026-06-20 02:26:23 on dist-test-slave-9vg1"
server_key: "bcc9c1868cde71c1b7399da0d54afd75"
server_key_iv: "dba0ace1487e80cd294ae05f519b9cc0"
server_key_version: "encryptionkey@0"
I20260620 02:26:23.373610 29083 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:26:23.378113 29097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:23.379909 29083 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:26:23.380182 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "29c8ee8609bb45b4aef795e00ee504a5"
format_stamp: "Formatted at 2026-06-20 02:26:23 on dist-test-slave-9vg1"
server_key: "bcc9c1868cde71c1b7399da0d54afd75"
server_key_iv: "dba0ace1487e80cd294ae05f519b9cc0"
server_key_version: "encryptionkey@0"
I20260620 02:26:23.380450 29083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:23.411847 29083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:23.412860 29083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:23.413223 29083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:23.414463 29083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:23.416133 29083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:23.416270 29083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:23.416421 29083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:23.416518 29083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:23.453022 29083 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:36245
I20260620 02:26:23.453081 29209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:36245 every 8 connection(s)
I20260620 02:26:23.454519 29083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:26:23.460481 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 29083
I20260620 02:26:23.460685 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:26:23.461226 10141 external_mini_cluster.cc:1428] Setting key 96e3ebaca6f45beb9d13b78aff60d75f
I20260620 02:26:23.464226 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39627
--builtin_ntp_servers=127.9.231.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:26:23.468636 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:23.457322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:59603 (local address 127.9.231.126:39627)
0620 02:26:23.457826 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:23.457831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:23.458971 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 02:26:23.461377 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:23.461385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:23.461482 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:23.461594 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:23.462812 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.463707 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.465456 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.465867 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.466016 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:23.467677 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:23.467682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:23.467685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:23.467706 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:23.467747 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:23.467758 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:23.467859 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:23.468124 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 02:26:23.468258 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":263,"threads_started":1}
I20260620 02:26:23.469738 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:23.457843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39627 (local address 127.9.231.65:59603)
0620 02:26:23.458793 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:23.458831 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:23.460976 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:23.461910 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:23.461919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:23.462018 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:23.462618 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:23.462638 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.463893 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.463897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:23.465276 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:23.465290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.466123 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.466130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:23.466395 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:23.467255 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:23.467258 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:23.467308 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:23.467459 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:23.467860 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:23.467874 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:23.467885 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:26:23.468099 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 02:26:23.468556 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":308,"threads_started":1}
I20260620 02:26:23.471694 29210 heartbeater.cc:347] Connected to a master server at 127.9.231.126:39627
I20260620 02:26:23.472105 29210 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:23.473017 29210 heartbeater.cc:511] Master 127.9.231.126:39627 requested a full tablet report, sending...
I20260620 02:26:23.475757 29024 ts_manager.cc:194] Registered new tserver with Master: 29c8ee8609bb45b4aef795e00ee504a5 (127.9.231.65:36245)
I20260620 02:26:23.477561 29024 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.65:59603
W20260620 02:26:23.817505 29214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:23.817874 29214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:23.818048 29214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:23.825907 29214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:23.826087 29214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:23.826278 29214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:26:23.836196 29214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:23.837618 29214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:23.839516 29214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:23.854172 29220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:23.854122 29222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:23.854538 29219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:23.855515 29214 server_base.cc:1034] running on GCE node
I20260620 02:26:23.856227 29214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:23.857383 29214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:23.858630 29214 hybrid_clock.cc:648] HybridClock initialized: now 1781922383858552 us; error 81 us; skew 500 ppm
I20260620 02:26:23.858992 29214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:23.861764 29214 webserver.cc:460] Webserver started at http://127.9.231.66:46569/ using document root <none> and password file <none>
I20260620 02:26:23.862565 29214 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:23.862725 29214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:23.863142 29214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:23.866212 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "be94ba6cff4047ddb3e6dbf6681615e9"
format_stamp: "Formatted at 2026-06-20 02:26:23 on dist-test-slave-9vg1"
server_key: "ddb0ba1709043903b2dab19ea53394f0"
server_key_iv: "1f6ee355dd70675f0c922b42dc7e1306"
server_key_version: "encryptionkey@0"
I20260620 02:26:23.867180 29214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "be94ba6cff4047ddb3e6dbf6681615e9"
format_stamp: "Formatted at 2026-06-20 02:26:23 on dist-test-slave-9vg1"
server_key: "ddb0ba1709043903b2dab19ea53394f0"
server_key_iv: "1f6ee355dd70675f0c922b42dc7e1306"
server_key_version: "encryptionkey@0"
I20260620 02:26:23.873221 29214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 02:26:23.877470 29228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:23.878999 29214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:26:23.879271 29214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "be94ba6cff4047ddb3e6dbf6681615e9"
format_stamp: "Formatted at 2026-06-20 02:26:23 on dist-test-slave-9vg1"
server_key: "ddb0ba1709043903b2dab19ea53394f0"
server_key_iv: "1f6ee355dd70675f0c922b42dc7e1306"
server_key_version: "encryptionkey@0"
I20260620 02:26:23.879487 29214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:23.920465 29214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:23.921638 29214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:23.921984 29214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:23.923130 29214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:23.924762 29214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:23.924896 29214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:23.925035 29214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:23.925132 29214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:23.960361 29214 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:41303
I20260620 02:26:23.960426 29340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:41303 every 8 connection(s)
I20260620 02:26:23.961784 29214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:26:23.965211 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 29214
I20260620 02:26:23.965406 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:26:23.966130 10141 external_mini_cluster.cc:1428] Setting key f79a903d232e132998f09bb48f19beda
I20260620 02:26:23.969439 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39627
--builtin_ntp_servers=127.9.231.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:26:23.976614 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:23.964668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:53847 (local address 127.9.231.126:39627)
0620 02:26:23.964876 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:23.964881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:23.966589 (+ 1708us) server_negotiation.cc:407] Connection header received
0620 02:26:23.968978 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:23.968984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:23.969082 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:23.969207 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:23.970655 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.971510 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.973200 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.973617 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.973766 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:23.975630 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:23.975635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:23.975638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:23.975666 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:23.975730 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:23.975749 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:23.975877 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:23.976155 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 02:26:23.976273 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:26:23.978012 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:23.965284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39627 (local address 127.9.231.66:53847)
0620 02:26:23.966403 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:23.966447 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:23.968627 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:23.969463 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:23.969473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:23.969645 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:23.970444 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:23.970476 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.971694 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.971700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:23.973021 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:23.973034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:23.973879 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:23.973886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:23.974147 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:23.974964 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:23.974967 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:23.975012 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:23.975407 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:23.975900 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:23.975911 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:23.975920 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:26:23.976140 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 02:26:23.976640 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":364,"threads_started":1}
I20260620 02:26:23.979622 29341 heartbeater.cc:347] Connected to a master server at 127.9.231.126:39627
I20260620 02:26:23.979964 29341 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:23.980783 29341 heartbeater.cc:511] Master 127.9.231.126:39627 requested a full tablet report, sending...
I20260620 02:26:23.982466 29024 ts_manager.cc:194] Registered new tserver with Master: be94ba6cff4047ddb3e6dbf6681615e9 (127.9.231.66:41303)
I20260620 02:26:23.983690 29024 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.66:53847
W20260620 02:26:24.324723 29344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:24.325035 29344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:24.325166 29344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:24.333081 29344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:24.333243 29344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:24.333448 29344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:26:24.343639 29344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:39627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:24.344929 29344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:24.346670 29344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:24.361100 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:24.361056 29352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:24.361123 29350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:24.362432 29344 server_base.cc:1034] running on GCE node
I20260620 02:26:24.363193 29344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:24.364252 29344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:24.365535 29344 hybrid_clock.cc:648] HybridClock initialized: now 1781922384365484 us; error 54 us; skew 500 ppm
I20260620 02:26:24.365909 29344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:24.368607 29344 webserver.cc:460] Webserver started at http://127.9.231.67:33643/ using document root <none> and password file <none>
I20260620 02:26:24.369343 29344 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:24.369482 29344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:24.369818 29344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:24.372900 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174"
format_stamp: "Formatted at 2026-06-20 02:26:24 on dist-test-slave-9vg1"
server_key: "3fe19a88d2e8456d9d66b10fafe54129"
server_key_iv: "d6e969b700e70d64dca3c436e44ec132"
server_key_version: "encryptionkey@0"
I20260620 02:26:24.373762 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174"
format_stamp: "Formatted at 2026-06-20 02:26:24 on dist-test-slave-9vg1"
server_key: "3fe19a88d2e8456d9d66b10fafe54129"
server_key_iv: "d6e969b700e70d64dca3c436e44ec132"
server_key_version: "encryptionkey@0"
I20260620 02:26:24.379619 29344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 02:26:24.383744 29358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:24.385340 29344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:26:24.385558 29344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174"
format_stamp: "Formatted at 2026-06-20 02:26:24 on dist-test-slave-9vg1"
server_key: "3fe19a88d2e8456d9d66b10fafe54129"
server_key_iv: "d6e969b700e70d64dca3c436e44ec132"
server_key_version: "encryptionkey@0"
I20260620 02:26:24.385860 29344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:24.410261 29344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:24.411257 29344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:24.411586 29344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:24.412617 29344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:24.414147 29344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:24.414286 29344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:24.414427 29344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:24.414525 29344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:24.449676 29344 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:33933
I20260620 02:26:24.449774 29470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:33933 every 8 connection(s)
I20260620 02:26:24.451220 29344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:26:24.459242 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 29344
I20260620 02:26:24.459432 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:26:24.459895 10141 external_mini_cluster.cc:1428] Setting key 15cbb0a2f8c26f47b74c9b2585cf6b03
I20260620 02:26:24.465026 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.453992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:43981 (local address 127.9.231.126:39627)
0620 02:26:24.454183 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.454188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.455868 (+ 1680us) server_negotiation.cc:407] Connection header received
0620 02:26:24.457716 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.457721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.457819 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.457935 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:24.459132 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.460048 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.461956 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.462430 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.462562 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.464156 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:24.464163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:24.464167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:24.464194 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:24.464225 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:24.464237 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:24.464331 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.464575 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.464688 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:26:24.466116 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.454636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:39627 (local address 127.9.231.67:43981)
0620 02:26:24.455700 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:24.455736 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:24.457425 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:24.458163 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:24.458170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:24.458262 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:24.458865 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.458890 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.460228 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.460235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.461793 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.461807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.462663 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.462673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.462949 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.463797 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:24.463800 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:24.463856 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:24.463963 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:24.464347 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:24.464359 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:24.464367 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:26:24.464561 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 02:26:24.464980 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":379,"threads_started":1}
I20260620 02:26:24.467796 29471 heartbeater.cc:347] Connected to a master server at 127.9.231.126:39627
I20260620 02:26:24.468180 29471 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:24.469115 29471 heartbeater.cc:511] Master 127.9.231.126:39627 requested a full tablet report, sending...
I20260620 02:26:24.470664 29024 ts_manager.cc:194] Registered new tserver with Master: 5ccd4cbcc57c4ef5aa5c4c432980e174 (127.9.231.67:33933)
I20260620 02:26:24.471604 29024 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.67:43981
I20260620 02:26:24.476156 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:26:24.481909 29210 heartbeater.cc:503] Master 127.9.231.126:39627 was elected leader, sending a full tablet report...
I20260620 02:26:24.487978 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.480936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.9.231.126:39627)
0620 02:26:24.481206 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.481215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.481529 (+ 314us) server_negotiation.cc:407] Connection header received
0620 02:26:24.481681 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.481686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.481772 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.481890 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:24.482504 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.483491 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.485064 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.485618 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.485775 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.487043 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:24.487048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:24.487102 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:24.487136 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:24.487173 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:24.487188 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:24.487371 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.487609 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.487674 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:26:24.492338 29024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:26:24.499769 29024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:26:24.525127 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.513643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:36245 (local address 127.0.0.1:41838)
0620 02:26:24.514526 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:24.514568 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:24.514763 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:24.516889 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:24.516897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:24.516934 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:24.517453 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.517470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.519872 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.519878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.521799 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.521815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.524007 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.524013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.524295 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.524365 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 02:26:24.524511 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:26:24.524726 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":11,"thread_start_us":2004,"threads_started":1}
I20260620 02:26:24.525214 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.513894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.9.231.65:36245)
0620 02:26:24.516113 (+ 2219us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.516123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.516191 (+ 68us) server_negotiation.cc:407] Connection header received
0620 02:26:24.516329 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.516340 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.516652 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.517738 (+ 1086us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:24.517997 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.519707 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.522027 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.523722 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.523919 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.524222 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.524677 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.524855 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2004,"thread_start_us":1875,"threads_started":1}
I20260620 02:26:24.527354 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.513836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:33933 (local address 127.0.0.1:52780)
0620 02:26:24.516386 (+ 2550us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:24.516404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:24.516525 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:24.517530 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:24.517534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:24.517563 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:24.517906 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.517918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.519872 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.519878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.524372 (+ 4494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.524384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.526414 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.526421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.526609 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.526645 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:26:24.526748 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 02:26:24.526898 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2405,"mutex_wait_us":55,"thread_start_us":2251,"threads_started":1}
I20260620 02:26:24.527451 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.514545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.9.231.67:33933)
0620 02:26:24.515509 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.515522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.516653 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 02:26:24.516757 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.516767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.516989 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.517221 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:24.518103 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.519707 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.524658 (+ 4951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.526189 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.526337 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.526605 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.526813 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.526991 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":601,"threads_started":1}
I20260620 02:26:24.529495 29145 tablet_service.cc:1467] Processing CreateTablet for tablet 093da5314afe413b9d614b15e99f2608 (DEFAULT_TABLE table=test-table [id=a0a37f8cbc2140b3abec04563aceb2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:24.531143 29145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093da5314afe413b9d614b15e99f2608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:24.531283 29406 tablet_service.cc:1467] Processing CreateTablet for tablet 093da5314afe413b9d614b15e99f2608 (DEFAULT_TABLE table=test-table [id=a0a37f8cbc2140b3abec04563aceb2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:24.532582 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.516962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.9.231.66:41303)
0620 02:26:24.520407 (+ 3445us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.520416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.523689 (+ 3273us) server_negotiation.cc:407] Connection header received
0620 02:26:24.523810 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.523820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.524132 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.524360 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:24.525027 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.526930 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.529877 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.531405 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.531553 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.531801 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.532034 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.532220 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3261,"thread_start_us":3137,"threads_started":1}
I20260620 02:26:24.532581 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.516678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:41303 (local address 127.0.0.1:50846)
0620 02:26:24.523373 (+ 6695us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:24.523394 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:24.523530 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:24.524370 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:24.524375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:24.524407 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:24.524819 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.524831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.527122 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.527129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.529637 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.529655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.531625 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.531631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.531818 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.531855 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:26:24.532065 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 02:26:24.532232 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6544,"thread_start_us":6995,"threads_started":1}
I20260620 02:26:24.532852 29406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093da5314afe413b9d614b15e99f2608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:24.536841 29276 tablet_service.cc:1467] Processing CreateTablet for tablet 093da5314afe413b9d614b15e99f2608 (DEFAULT_TABLE table=test-table [id=a0a37f8cbc2140b3abec04563aceb2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:24.538314 29276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093da5314afe413b9d614b15e99f2608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:24.543756 29489 tablet_bootstrap.cc:492] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Bootstrap starting.
I20260620 02:26:24.547158 29489 tablet_bootstrap.cc:654] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:24.548668 29489 log.cc:824] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:24.549047 29490 tablet_bootstrap.cc:492] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Bootstrap starting.
I20260620 02:26:24.552819 29489 tablet_bootstrap.cc:492] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: No bootstrap required, opened a new log
I20260620 02:26:24.553130 29489 ts_tablet_manager.cc:1397] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 02:26:24.553365 29490 tablet_bootstrap.cc:654] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:24.554414 29492 tablet_bootstrap.cc:492] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Bootstrap starting.
I20260620 02:26:24.554858 29490 log.cc:824] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:24.558254 29490 tablet_bootstrap.cc:492] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: No bootstrap required, opened a new log
I20260620 02:26:24.558386 29492 tablet_bootstrap.cc:654] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:24.558559 29490 ts_tablet_manager.cc:1397] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 02:26:24.559742 29492 log.cc:824] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:24.562992 29492 tablet_bootstrap.cc:492] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: No bootstrap required, opened a new log
I20260620 02:26:24.562772 29489 raft_consensus.cc:348] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.563337 29492 ts_tablet_manager.cc:1397] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:26:24.563338 29489 raft_consensus.cc:374] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:24.563439 29489 raft_consensus.cc:729] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c8ee8609bb45b4aef795e00ee504a5, State: Initialized, Role: FOLLOWER
I20260620 02:26:24.564105 29489 consensus_queue.cc:260] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.565915 29489 ts_tablet_manager.cc:1428] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 02:26:24.569195 29490 raft_consensus.cc:348] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.569644 29490 raft_consensus.cc:374] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:24.569743 29490 raft_consensus.cc:729] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccd4cbcc57c4ef5aa5c4c432980e174, State: Initialized, Role: FOLLOWER
I20260620 02:26:24.570488 29490 consensus_queue.cc:260] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.571667 29471 heartbeater.cc:503] Master 127.9.231.126:39627 was elected leader, sending a full tablet report...
I20260620 02:26:24.572377 29490 ts_tablet_manager.cc:1428] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260620 02:26:24.574054 29492 raft_consensus.cc:348] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.574572 29492 raft_consensus.cc:374] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:24.574708 29492 raft_consensus.cc:729] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be94ba6cff4047ddb3e6dbf6681615e9, State: Initialized, Role: FOLLOWER
I20260620 02:26:24.575362 29492 consensus_queue.cc:260] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.576581 29341 heartbeater.cc:503] Master 127.9.231.126:39627 was elected leader, sending a full tablet report...
I20260620 02:26:24.577412 29492 ts_tablet_manager.cc:1428] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260620 02:26:24.703245 29472 tablet.cc:2369] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:26:24.707283 29211 tablet.cc:2369] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:26:24.714278 29342 tablet.cc:2369] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:24.808509 29497 raft_consensus.cc:482] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:26:24.808849 29497 raft_consensus.cc:504] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.810925 29497 leader_election.cc:288] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29c8ee8609bb45b4aef795e00ee504a5 (127.9.231.65:36245), 5ccd4cbcc57c4ef5aa5c4c432980e174 (127.9.231.67:33933)
I20260620 02:26:24.819413 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.811840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:60143 (local address 127.9.231.67:33933)
0620 02:26:24.812035 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.812042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.812067 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:26:24.812261 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.812268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.812343 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.812408 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:24.813288 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.814207 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.816940 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.818234 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.818361 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.818428 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.818983 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.819165 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:26:24.819494 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.811274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:33933 (local address 127.9.231.66:60143)
0620 02:26:24.811884 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:24.811906 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:24.812052 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:24.812630 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:24.812635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:24.812666 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:24.813089 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.813101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.814337 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.814341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.816701 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.816718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.818472 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.818477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.818720 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.818798 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 02:26:24.818895 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:26:24.819132 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":127,"thread_start_us":797,"threads_started":1}
I20260620 02:26:24.819535 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.811828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:59977 (local address 127.9.231.65:36245)
0620 02:26:24.812035 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.812042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.813018 (+ 976us) server_negotiation.cc:407] Connection header received
0620 02:26:24.813240 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.813247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.813338 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.813451 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:24.813995 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.814787 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.817074 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.818055 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.818230 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.818303 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.819158 (+ 855us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.819305 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:26:24.820153 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.811366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:36245 (local address 127.9.231.66:59977)
0620 02:26:24.812900 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:24.812916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:24.813094 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:24.813473 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:24.813478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:24.813503 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:24.813832 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.813842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.814942 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.814947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.816892 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:24.816907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.818460 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.818467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:24.818720 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.818806 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 02:26:24.818895 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:26:24.819793 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1415,"mutex_wait_us":719,"thread_start_us":1157,"threads_started":1}
I20260620 02:26:24.820919 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093da5314afe413b9d614b15e99f2608" candidate_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" is_pre_election: true
I20260620 02:26:24.821215 29165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093da5314afe413b9d614b15e99f2608" candidate_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c8ee8609bb45b4aef795e00ee504a5" is_pre_election: true
I20260620 02:26:24.821506 29426 raft_consensus.cc:2454] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be94ba6cff4047ddb3e6dbf6681615e9 in term 0.
I20260620 02:26:24.821707 29165 raft_consensus.cc:2454] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be94ba6cff4047ddb3e6dbf6681615e9 in term 0.
I20260620 02:26:24.822321 29232 leader_election.cc:302] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29c8ee8609bb45b4aef795e00ee504a5, 5ccd4cbcc57c4ef5aa5c4c432980e174, be94ba6cff4047ddb3e6dbf6681615e9; no voters:
I20260620 02:26:24.822820 29497 raft_consensus.cc:2781] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:26:24.822979 29497 raft_consensus.cc:482] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:26:24.823091 29497 raft_consensus.cc:3037] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:24.826516 29497 raft_consensus.cc:504] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.827299 29497 leader_election.cc:288] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [CANDIDATE]: Term 1 election: Requested vote from peers 29c8ee8609bb45b4aef795e00ee504a5 (127.9.231.65:36245), 5ccd4cbcc57c4ef5aa5c4c432980e174 (127.9.231.67:33933)
I20260620 02:26:24.827925 29165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093da5314afe413b9d614b15e99f2608" candidate_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c8ee8609bb45b4aef795e00ee504a5"
I20260620 02:26:24.828024 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093da5314afe413b9d614b15e99f2608" candidate_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174"
I20260620 02:26:24.828182 29165 raft_consensus.cc:3037] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:24.828254 29426 raft_consensus.cc:3037] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:24.830637 29165 raft_consensus.cc:2454] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be94ba6cff4047ddb3e6dbf6681615e9 in term 1.
I20260620 02:26:24.831220 29426 raft_consensus.cc:2454] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be94ba6cff4047ddb3e6dbf6681615e9 in term 1.
I20260620 02:26:24.831249 29229 leader_election.cc:302] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c8ee8609bb45b4aef795e00ee504a5, be94ba6cff4047ddb3e6dbf6681615e9; no voters:
I20260620 02:26:24.831642 29497 raft_consensus.cc:2781] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:24.832154 29497 raft_consensus.cc:686] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 1 LEADER]: Becoming Leader. State: Replica: be94ba6cff4047ddb3e6dbf6681615e9, State: Running, Role: LEADER
I20260620 02:26:24.832734 29497 consensus_queue.cc:237] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } }
I20260620 02:26:24.837666 29023 catalog_manager.cc:5495] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to be94ba6cff4047ddb3e6dbf6681615e9 (127.9.231.66). New cstate: current_term: 1 leader_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be94ba6cff4047ddb3e6dbf6681615e9" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 41303 } health_report { overall_health: HEALTHY } } }
I20260620 02:26:24.903285 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.897057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.9.231.66:41303)
0620 02:26:24.897310 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.897317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.897367 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:26:24.897662 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.897666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.897769 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.897942 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:26:24.898346 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.899227 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.900666 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.901118 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.901264 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.902004 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:26:24.902553 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:26:24.902769 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.902881 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.902926 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 02:26:24.912034 29426 raft_consensus.cc:1261] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 1 FOLLOWER]: Refusing update from remote peer be94ba6cff4047ddb3e6dbf6681615e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:24.912030 29165 raft_consensus.cc:1261] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 1 FOLLOWER]: Refusing update from remote peer be94ba6cff4047ddb3e6dbf6681615e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:24.913125 29500 consensus_queue.cc:1035] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c8ee8609bb45b4aef795e00ee504a5" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 36245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:24.913501 29497 consensus_queue.cc:1035] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccd4cbcc57c4ef5aa5c4c432980e174" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 33933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:24.926800 29504 mvcc.cc:205] Tried to move back new op lower bound from 7298754088589103104 to 7298754088279449600. Current Snapshot: MvccSnapshot[applied={T|T < 7298754088589103104}]
I20260620 02:26:24.938367 29023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:24.938704 29023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:24.942926 29023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dad538b5cca94e309ddfd3e548d1c4f4: Sending DeleteTablet for 3 replicas of tablet 093da5314afe413b9d614b15e99f2608
I20260620 02:26:24.944375 29145 tablet_service.cc:1514] Processing DeleteTablet for tablet 093da5314afe413b9d614b15e99f2608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:24 UTC) from {username='slave'} at 127.0.0.1:41838
I20260620 02:26:24.944375 29406 tablet_service.cc:1514] Processing DeleteTablet for tablet 093da5314afe413b9d614b15e99f2608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:24 UTC) from {username='slave'} at 127.0.0.1:52780
I20260620 02:26:24.944875 29276 tablet_service.cc:1514] Processing DeleteTablet for tablet 093da5314afe413b9d614b15e99f2608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:24 UTC) from {username='slave'} at 127.0.0.1:50846
I20260620 02:26:24.945605 29514 tablet_replica.cc:330] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: stopping tablet replica
I20260620 02:26:24.946273 29513 tablet_replica.cc:330] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: stopping tablet replica
I20260620 02:26:24.946401 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 29083
I20260620 02:26:24.946555 29515 tablet_replica.cc:330] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: stopping tablet replica
I20260620 02:26:24.946987 29515 raft_consensus.cc:2229] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:26:24.946977 29513 raft_consensus.cc:2229] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:24.947506 29514 raft_consensus.cc:2229] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:24.947772 29515 raft_consensus.cc:2258] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:24.948312 29513 raft_consensus.cc:2258] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:24.948678 29514 raft_consensus.cc:2258] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:24.950382 29515 ts_tablet_manager.cc:1905] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:24.954303 29514 ts_tablet_manager.cc:1905] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:24.952760 29513 ts_tablet_manager.cc:1905] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:24.960414 29515 ts_tablet_manager.cc:1918] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:24.960621 29515 log.cc:1196] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/093da5314afe413b9d614b15e99f2608
I20260620 02:26:24.961258 29515 ts_tablet_manager.cc:1939] T 093da5314afe413b9d614b15e99f2608 P be94ba6cff4047ddb3e6dbf6681615e9: Deleting consensus metadata
I20260620 02:26:24.963313 29008 catalog_manager.cc:4841] TS be94ba6cff4047ddb3e6dbf6681615e9 (127.9.231.66:41303): tablet 093da5314afe413b9d614b15e99f2608 (table test-table [id=a0a37f8cbc2140b3abec04563aceb2f9]) successfully deleted
I20260620 02:26:24.970873 29514 ts_tablet_manager.cc:1918] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:24.971127 29514 log.cc:1196] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/093da5314afe413b9d614b15e99f2608
I20260620 02:26:24.971275 29513 ts_tablet_manager.cc:1918] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:24.971449 29513 log.cc:1196] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/093da5314afe413b9d614b15e99f2608
I20260620 02:26:24.971748 29514 ts_tablet_manager.cc:1939] T 093da5314afe413b9d614b15e99f2608 P 5ccd4cbcc57c4ef5aa5c4c432980e174: Deleting consensus metadata
I20260620 02:26:24.971964 29513 ts_tablet_manager.cc:1939] T 093da5314afe413b9d614b15e99f2608 P 29c8ee8609bb45b4aef795e00ee504a5: Deleting consensus metadata
I20260620 02:26:24.973978 29011 catalog_manager.cc:4841] TS 5ccd4cbcc57c4ef5aa5c4c432980e174 (127.9.231.67:33933): tablet 093da5314afe413b9d614b15e99f2608 (table test-table [id=a0a37f8cbc2140b3abec04563aceb2f9]) successfully deleted
I20260620 02:26:24.974944 29008 catalog_manager.cc:4841] TS 29c8ee8609bb45b4aef795e00ee504a5 (127.9.231.65:36245): tablet 093da5314afe413b9d614b15e99f2608 (table test-table [id=a0a37f8cbc2140b3abec04563aceb2f9]) successfully deleted
I20260620 02:26:24.975740 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:24.949030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.9.231.65:36245)
0620 02:26:24.949227 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:24.949235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:24.959536 (+ 10301us) server_negotiation.cc:407] Connection header received
0620 02:26:24.959842 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:24.959848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:24.959981 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:24.960070 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:24.960838 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.961931 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.964957 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:24.965594 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:24.965703 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:24.973708 (+ 8005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:24.973719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:24.973739 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:24.974538 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:24.974606 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:24.974641 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:24.974756 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:24.975365 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 02:26:24.975474 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:26:24.976512 29205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:25.070655 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 29083
I20260620 02:26:25.090476 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 29214
I20260620 02:26:25.099431 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:25.091604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.9.231.66:41303)
0620 02:26:25.091818 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:25.091825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:25.091847 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:26:25.091930 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:25.091940 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:25.092047 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:25.092177 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:25.092561 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:25.093761 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:25.095447 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:25.096239 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:25.096387 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:25.097538 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:25.097551 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:25.097573 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:25.098444 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:25.098509 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:25.098536 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:25.098659 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:25.098856 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 02:26:25.098981 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:26:25.100250 29336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:25.218276 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 29214
I20260620 02:26:25.238448 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 29344
I20260620 02:26:25.246325 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:25.239422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.9.231.67:33933)
0620 02:26:25.239697 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:25.239702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:25.239723 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:26:25.239889 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:25.239894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:25.239993 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:25.240100 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:25.240781 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:25.241523 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:25.242944 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:25.243475 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:25.243587 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:25.244825 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:25.244835 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:25.244851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:25.245436 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:25.245488 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:25.245509 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:25.245600 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:25.245810 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 02:26:25.245908 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:26:25.247164 29466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:25.341603 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 29344
I20260620 02:26:25.360407 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 28993
I20260620 02:26:25.361826 29054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:25.477659 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 28993
2026-06-20T02:26:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3203 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:25 dist-test-slave-9vg1 krb5kdc[29523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:25 dist-test-slave-9vg1 krb5kdc[29523](info): set up 2 sockets
Jun 20 02:26:25 dist-test-slave-9vg1 krb5kdc[29523](info): commencing operation
krb5kdc: starting...
W20260620 02:26:26.656940 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:26 dist-test-slave-9vg1 krb5kdc[29523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:26:26.943428 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33907
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:27.301311 29536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:27.301739 29536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:27.301916 29536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:27.309631 29536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:27.309816 29536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:27.309929 29536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:27.310043 29536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:27.310132 29536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:27.321173 29536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:27.322865 29536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:27.324496 29536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:27.339390 29544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:27.339455 29541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:27.339704 29542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:27.340525 29536 server_base.cc:1034] running on GCE node
I20260620 02:26:27.341578 29536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:27.343348 29536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:27.344631 29536 hybrid_clock.cc:648] HybridClock initialized: now 1781922387344608 us; error 65 us; skew 500 ppm
I20260620 02:26:27.345055 29536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:27 dist-test-slave-9vg1 krb5kdc[29523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922387, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:27.348745 29536 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:26:27.350630 29536 webserver.cc:460] Webserver started at http://127.9.231.126:36739/ using document root <none> and password file <none>
I20260620 02:26:27.351506 29536 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:27.351662 29536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:27.351992 29536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:27.355439 29536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "b80f1140e2c341ffb8dc7d5cd2370d69"
format_stamp: "Formatted at 2026-06-20 02:26:27 on dist-test-slave-9vg1"
server_key: "7ca6eb071bbf9064c54179706eacd8ce"
server_key_iv: "ad678c56b60d86ee75255edb5b13151a"
server_key_version: "encryptionkey@0"
I20260620 02:26:27.356343 29536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "b80f1140e2c341ffb8dc7d5cd2370d69"
format_stamp: "Formatted at 2026-06-20 02:26:27 on dist-test-slave-9vg1"
server_key: "7ca6eb071bbf9064c54179706eacd8ce"
server_key_iv: "ad678c56b60d86ee75255edb5b13151a"
server_key_version: "encryptionkey@0"
I20260620 02:26:27.362571 29536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:26:27.366779 29551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:27.368561 29536 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:26:27.368824 29536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "b80f1140e2c341ffb8dc7d5cd2370d69"
format_stamp: "Formatted at 2026-06-20 02:26:27 on dist-test-slave-9vg1"
server_key: "7ca6eb071bbf9064c54179706eacd8ce"
server_key_iv: "ad678c56b60d86ee75255edb5b13151a"
server_key_version: "encryptionkey@0"
I20260620 02:26:27.369072 29536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:27.398057 29536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:27.403087 29536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:27.403450 29536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:27.423231 29536 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33907
I20260620 02:26:27.423265 29602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33907 every 8 connection(s)
I20260620 02:26:27.424782 29536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:27.428857 29603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:27.431723 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 29536
I20260620 02:26:27.431941 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:27.432477 10141 external_mini_cluster.cc:1428] Setting key 568cc12d3195ba4eef6b535a4486f2e4
I20260620 02:26:27.441923 29603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69: Bootstrap starting.
I20260620 02:26:27.446116 29603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:27.447641 29603 log.cc:824] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:26:27 dist-test-slave-9vg1 krb5kdc[29523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:27.451529 29603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69: No bootstrap required, opened a new log
I20260620 02:26:27.459465 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:27.434712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.9.231.126:33907)
0620 02:26:27.435627 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:27.435640 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:27.435695 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:26:27.437432 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:27.437464 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:27.437957 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:27.438334 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:27.440342 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:27.441776 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:27.443739 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:27.444461 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:27.444608 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:27.450266 (+ 5658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:27.450276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:27.450293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:27.450335 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:27.454003 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:27.455219 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:27.455228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:27.455236 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:27.455552 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:27.456102 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:27.456107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:27.456109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:27.456820 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:27.457139 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:27.457683 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 02:26:27.458200 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":443,"threads_started":1}
I20260620 02:26:27.461663 29603 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33907 } }
I20260620 02:26:27.462100 29603 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:27.462232 29603 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80f1140e2c341ffb8dc7d5cd2370d69, State: Initialized, Role: FOLLOWER
I20260620 02:26:27.462898 29603 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33907 } }
I20260620 02:26:27.463255 29603 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:27.463404 29603 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:27.463594 29603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:27.465953 29603 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33907 } }
I20260620 02:26:27.466727 29603 leader_election.cc:302] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80f1140e2c341ffb8dc7d5cd2370d69; no voters:
I20260620 02:26:27.467489 29603 leader_election.cc:288] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:27.467679 29608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:27.468469 29608 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [term 1 LEADER]: Becoming Leader. State: Replica: b80f1140e2c341ffb8dc7d5cd2370d69, State: Running, Role: LEADER
I20260620 02:26:27.469098 29608 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33907 } }
I20260620 02:26:27.469702 29603 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:27.472303 29609 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33907 } } }
I20260620 02:26:27.472648 29610 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80f1140e2c341ffb8dc7d5cd2370d69. Latest consensus state: current_term: 1 leader_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80f1140e2c341ffb8dc7d5cd2370d69" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33907 } } }
I20260620 02:26:27.472955 29609 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:27.472983 29610 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:27.475015 29617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:27.480818 29617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:27.491385 29617 catalog_manager.cc:1347] Generated new cluster ID: 5a5e1987b115477daab98ec17d35d649
I20260620 02:26:27.491561 29617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:27.505789 29617 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:27.507663 29617 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:27.524811 29617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69: Generated new TSK 0
I20260620 02:26:27.525763 29617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.231.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
Entry for principal kudu/127.9.231.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab.
I20260620 02:26:27.603955 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33907
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:27.969431 29629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:27.969870 29629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:27.970049 29629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:27.978353 29629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:27.978540 29629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:27.978757 29629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:26:27.989279 29629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.65.keytab
--principal=kudu/127.9.231.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:27.990808 29629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:27.992280 29629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:28.007414 29634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:28.007386 29635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:28.007436 29637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:28.008953 29629 server_base.cc:1034] running on GCE node
I20260620 02:26:28.009794 29629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:28.010955 29629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:28.012238 29629 hybrid_clock.cc:648] HybridClock initialized: now 1781922388012188 us; error 38 us; skew 500 ppm
I20260620 02:26:28.012703 29629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:28 dist-test-slave-9vg1 krb5kdc[29523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:28.016331 29629 init.cc:377] Logged in from keytab as kudu/127.9.231.65@KRBTEST.COM (short username kudu)
I20260620 02:26:28.018217 29629 webserver.cc:460] Webserver started at http://127.9.231.65:38317/ using document root <none> and password file <none>
I20260620 02:26:28.019011 29629 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:28.019236 29629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:28.019596 29629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:28.022868 29629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "d43724228200446d8a8a6b704b8bcf3f"
format_stamp: "Formatted at 2026-06-20 02:26:28 on dist-test-slave-9vg1"
server_key: "1fac18307743ff1e147f88a5f187ca99"
server_key_iv: "6880bac6d2152acb610dccf65de8a447"
server_key_version: "encryptionkey@0"
I20260620 02:26:28.023836 29629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "d43724228200446d8a8a6b704b8bcf3f"
format_stamp: "Formatted at 2026-06-20 02:26:28 on dist-test-slave-9vg1"
server_key: "1fac18307743ff1e147f88a5f187ca99"
server_key_iv: "6880bac6d2152acb610dccf65de8a447"
server_key_version: "encryptionkey@0"
I20260620 02:26:28.030090 29629 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260620 02:26:28.034665 29644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:28.036556 29629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:26:28.036831 29629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "d43724228200446d8a8a6b704b8bcf3f"
format_stamp: "Formatted at 2026-06-20 02:26:28 on dist-test-slave-9vg1"
server_key: "1fac18307743ff1e147f88a5f187ca99"
server_key_iv: "6880bac6d2152acb610dccf65de8a447"
server_key_version: "encryptionkey@0"
I20260620 02:26:28.037098 29629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:28.076722 29629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:28.081444 29629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:28.081784 29629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:28.082841 29629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:28.084427 29629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:28.084539 29629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:28.084717 29629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:28.084793 29629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:28.122305 29629 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:38239
I20260620 02:26:28.122345 29756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:38239 every 8 connection(s)
I20260620 02:26:28.123962 29629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:26:28.133486 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 29629
I20260620 02:26:28.133684 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:26:28.134289 10141 external_mini_cluster.cc:1428] Setting key 3586321a5d69d5343e55a28fdbade0b3
Jun 20 02:26:28 dist-test-slave-9vg1 krb5kdc[29523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.65@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:28.147987 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:28.127003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:53383 (local address 127.9.231.126:33907)
0620 02:26:28.127537 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:28.127542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:28.128519 (+ 977us) server_negotiation.cc:407] Connection header received
0620 02:26:28.130440 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:28.130446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:28.130548 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:28.130683 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:28.132364 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.133206 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.135303 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.135793 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.135897 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:28.140133 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:28.140142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:28.140155 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:28.140206 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:28.144130 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:28.145127 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:28.145135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:28.145139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:28.145299 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:28.145881 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:28.145889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:28.145895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:28.146240 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:28.146359 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:28.147502 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0620 02:26:28.147716 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":297,"threads_started":1}
I20260620 02:26:28.149724 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:28.127621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33907 (local address 127.9.231.65:53383)
0620 02:26:28.128345 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:28.128389 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:28.130127 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:28.130938 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:28.130947 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:28.131562 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:28.132171 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:28.132190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.133405 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.133410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:28.135171 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:28.135186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.136054 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.136062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:28.136388 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:28.137150 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:28.137180 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:28.139815 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:28.144322 (+ 4507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:28.144330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:28.144346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:28.144966 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:28.145451 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:28.145455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:28.145457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:28.145734 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:28.146405 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:28.146416 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:28.146985 (+ 569us) client_negotiation.cc:769] Sending connection context
0620 02:26:28.147392 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 02:26:28.148131 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":295,"threads_started":1}
I20260620 02:26:28.151590 29757 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33907
I20260620 02:26:28.151968 29757 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:28.152779 29757 heartbeater.cc:511] Master 127.9.231.126:33907 requested a full tablet report, sending...
I20260620 02:26:28.155009 29568 ts_manager.cc:194] Registered new tserver with Master: d43724228200446d8a8a6b704b8bcf3f (127.9.231.65:38239)
I20260620 02:26:28.156596 29568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.65@KRBTEST.COM'} at 127.9.231.65:53383
WARNING: no policy specified for kudu/127.9.231.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
Entry for principal kudu/127.9.231.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab.
I20260620 02:26:28.199223 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33907
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:28.555398 29763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:28.555712 29763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:28.555872 29763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:28.563637 29763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:28.563760 29763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:28.563951 29763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:26:28.574568 29763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.66.keytab
--principal=kudu/127.9.231.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:28.576057 29763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:28.577417 29763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:28.591364 29768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:28.591388 29771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:28.591336 29769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:28.592192 29763 server_base.cc:1034] running on GCE node
I20260620 02:26:28.592864 29763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:28.593864 29763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:28.595178 29763 hybrid_clock.cc:648] HybridClock initialized: now 1781922388595104 us; error 45 us; skew 500 ppm
I20260620 02:26:28.595532 29763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:28 dist-test-slave-9vg1 krb5kdc[29523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:28.598860 29763 init.cc:377] Logged in from keytab as kudu/127.9.231.66@KRBTEST.COM (short username kudu)
I20260620 02:26:28.600641 29763 webserver.cc:460] Webserver started at http://127.9.231.66:44161/ using document root <none> and password file <none>
I20260620 02:26:28.601446 29763 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:28.601601 29763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:28.601936 29763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:28.605067 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "580f759dd1b241c68070a907ab919ba6"
format_stamp: "Formatted at 2026-06-20 02:26:28 on dist-test-slave-9vg1"
server_key: "1da9daa196bc91349392dd188e8e397d"
server_key_iv: "3fa0e103f605655a4cd6015747d78c30"
server_key_version: "encryptionkey@0"
I20260620 02:26:28.605893 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "580f759dd1b241c68070a907ab919ba6"
format_stamp: "Formatted at 2026-06-20 02:26:28 on dist-test-slave-9vg1"
server_key: "1da9daa196bc91349392dd188e8e397d"
server_key_iv: "3fa0e103f605655a4cd6015747d78c30"
server_key_version: "encryptionkey@0"
I20260620 02:26:28.611572 29763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:26:28.615422 29778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:28.617022 29763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:26:28.617241 29763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "580f759dd1b241c68070a907ab919ba6"
format_stamp: "Formatted at 2026-06-20 02:26:28 on dist-test-slave-9vg1"
server_key: "1da9daa196bc91349392dd188e8e397d"
server_key_iv: "3fa0e103f605655a4cd6015747d78c30"
server_key_version: "encryptionkey@0"
I20260620 02:26:28.617487 29763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:28.635985 29763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:28.640945 29763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:28.641319 29763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:28.642402 29763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:28.644039 29763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:28.644160 29763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:28.644322 29763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:28.644395 29763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:28.680641 29763 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:33741
I20260620 02:26:28.680740 29890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:33741 every 8 connection(s)
I20260620 02:26:28.682047 29763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:26:28.688092 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 29763
I20260620 02:26:28.688338 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:26:28.688800 10141 external_mini_cluster.cc:1428] Setting key 3783f08bbc96bb1eb9b8f732a4a41357
Jun 20 02:26:28 dist-test-slave-9vg1 krb5kdc[29523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.66@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:28.706559 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:28.685227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:53089 (local address 127.9.231.126:33907)
0620 02:26:28.685740 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:28.685749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:28.686945 (+ 1196us) server_negotiation.cc:407] Connection header received
0620 02:26:28.689089 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:28.689095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:28.689196 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:28.689338 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:28.691422 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.692596 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.694461 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.695226 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.695371 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:28.699666 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:28.699677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:28.699684 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:28.699735 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:28.702509 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:28.703710 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:28.703715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:28.703717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:28.703876 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:28.704489 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:28.704493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:28.704495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:28.704799 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:28.704948 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:28.706013 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0620 02:26:28.706266 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260620 02:26:28.708110 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:28.685854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33907 (local address 127.9.231.66:53089)
0620 02:26:28.686740 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:28.686793 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:28.688728 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:28.689783 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:28.689794 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:28.690399 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:28.691232 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:28.691253 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.692800 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.692805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:28.694283 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:28.694299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:28.695534 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:28.695541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:28.695767 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:28.696552 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:28.696629 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:28.699349 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:28.702701 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:28.702715 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:28.702740 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:28.703513 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:28.704021 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:28.704029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:28.704035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:28.704315 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:28.704965 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:28.704976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:28.705506 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 02:26:28.705867 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 02:26:28.706510 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":333,"threads_started":1}
I20260620 02:26:28.709793 29891 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33907
I20260620 02:26:28.710176 29891 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:28.711035 29891 heartbeater.cc:511] Master 127.9.231.126:33907 requested a full tablet report, sending...
I20260620 02:26:28.712845 29568 ts_manager.cc:194] Registered new tserver with Master: 580f759dd1b241c68070a907ab919ba6 (127.9.231.66:33741)
I20260620 02:26:28.713848 29568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.66@KRBTEST.COM'} at 127.9.231.66:53089
WARNING: no policy specified for kudu/127.9.231.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
Entry for principal kudu/127.9.231.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab.
I20260620 02:26:28.755771 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33907
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:29.109786 29897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:29.110200 29897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:29.110401 29897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:29.118265 29897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:29.118471 29897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:29.118667 29897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:26:29.132475 29897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.67.keytab
--principal=kudu/127.9.231.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:29.133893 29897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:29.135409 29897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:29.149547 29902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:29.149665 29903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:29.149720 29905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:29.150810 29897 server_base.cc:1034] running on GCE node
I20260620 02:26:29.151546 29897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:29.152609 29897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:29.153863 29897 hybrid_clock.cc:648] HybridClock initialized: now 1781922389153787 us; error 77 us; skew 500 ppm
I20260620 02:26:29.154220 29897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:29 dist-test-slave-9vg1 krb5kdc[29523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922389, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:29.157595 29897 init.cc:377] Logged in from keytab as kudu/127.9.231.67@KRBTEST.COM (short username kudu)
I20260620 02:26:29.159410 29897 webserver.cc:460] Webserver started at http://127.9.231.67:45393/ using document root <none> and password file <none>
I20260620 02:26:29.160070 29757 heartbeater.cc:503] Master 127.9.231.126:33907 was elected leader, sending a full tablet report...
I20260620 02:26:29.160164 29897 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:29.160293 29897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:29.160686 29897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:29.163957 29897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "73ce065722a94bc297cdbb80c2c0c8d5"
format_stamp: "Formatted at 2026-06-20 02:26:29 on dist-test-slave-9vg1"
server_key: "96ebcc655f6c9253150ebe78be99f1d1"
server_key_iv: "c4270dfcb764c437fd5aa0b7877d09a7"
server_key_version: "encryptionkey@0"
I20260620 02:26:29.164911 29897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "73ce065722a94bc297cdbb80c2c0c8d5"
format_stamp: "Formatted at 2026-06-20 02:26:29 on dist-test-slave-9vg1"
server_key: "96ebcc655f6c9253150ebe78be99f1d1"
server_key_iv: "c4270dfcb764c437fd5aa0b7877d09a7"
server_key_version: "encryptionkey@0"
I20260620 02:26:29.170735 29897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:26:29.174865 29912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:29.176426 29897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:29.176622 29897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "73ce065722a94bc297cdbb80c2c0c8d5"
format_stamp: "Formatted at 2026-06-20 02:26:29 on dist-test-slave-9vg1"
server_key: "96ebcc655f6c9253150ebe78be99f1d1"
server_key_iv: "c4270dfcb764c437fd5aa0b7877d09a7"
server_key_version: "encryptionkey@0"
I20260620 02:26:29.176842 29897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:29.190670 29897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:29.194849 29897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:29.195201 29897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:29.196183 29897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:29.197649 29897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:29.197727 29897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:29.197901 29897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:29.197952 29897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:29.232816 29897 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:37857
I20260620 02:26:29.232854 30024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:37857 every 8 connection(s)
I20260620 02:26:29.234184 29897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:26:29.244029 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 29897
I20260620 02:26:29.244241 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:26:29.244680 10141 external_mini_cluster.cc:1428] Setting key bcc1e64f7546b8793f24945294b3dbfb
Jun 20 02:26:29 dist-test-slave-9vg1 krb5kdc[29523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922389, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.67@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:26:29.256759 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.237112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:50675 (local address 127.9.231.126:33907)
0620 02:26:29.237572 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.237578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.238660 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 02:26:29.240801 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.240806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.240904 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.241059 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:29.242846 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.243756 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.245627 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.246069 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.246232 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.250274 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:29.250280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:29.250284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:29.250322 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:29.253297 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.254167 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.254175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.254183 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.254318 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.254784 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.254793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.254796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.255174 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:29.255337 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.256102 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.256348 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":205,"threads_started":1}
I20260620 02:26:29.257814 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.237739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:33907 (local address 127.9.231.67:50675)
0620 02:26:29.238480 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:29.238519 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:29.240381 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:29.241318 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:29.241327 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:29.241906 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:29.242625 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.242646 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.243952 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.243956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.245456 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.245477 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.246334 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.246341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.246601 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.247468 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:26:29.247497 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:29.250020 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:29.253490 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:29.253501 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:29.253523 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:29.254022 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:29.254444 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:26:29.254449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:26:29.254452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:26:29.254663 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:26:29.255309 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:29.255318 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:29.255701 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 02:26:29.256009 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 02:26:29.256591 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":301,"threads_started":1}
I20260620 02:26:29.259452 30025 heartbeater.cc:347] Connected to a master server at 127.9.231.126:33907
I20260620 02:26:29.259845 30025 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:29.260624 30025 heartbeater.cc:511] Master 127.9.231.126:33907 requested a full tablet report, sending...
I20260620 02:26:29.262166 29568 ts_manager.cc:194] Registered new tserver with Master: 73ce065722a94bc297cdbb80c2c0c8d5 (127.9.231.67:37857)
I20260620 02:26:29.263159 29568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.231.67@KRBTEST.COM'} at 127.9.231.67:50675
I20260620 02:26:29.273824 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:26:29.293574 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.278886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.9.231.126:33907)
0620 02:26:29.279136 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.279141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.279419 (+ 278us) server_negotiation.cc:407] Connection header received
0620 02:26:29.279603 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.279607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.279699 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.279796 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:29.281571 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.282362 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.284226 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.285019 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.285204 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.287843 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:29.287854 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:29.287857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:29.287898 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:29.290520 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.291423 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.291429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.291433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.291578 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.291990 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.291995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.291999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.292375 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:29.292642 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.292995 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.293248 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:26:29.298100 29568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:26:29.305992 29568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:26:29.343428 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.318863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:37857 (local address 127.0.0.1:35842)
0620 02:26:29.330753 (+ 11890us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:29.330822 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:29.331191 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:29.332855 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:29.332869 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:29.332922 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:29.333481 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.333502 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.335855 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.335865 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.339366 (+ 3501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.339395 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.342100 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.342108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.342427 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.342510 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 02:26:29.342664 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:26:29.342920 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11470,"thread_start_us":11216,"threads_started":1}
I20260620 02:26:29.344379 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.319421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.9.231.67:37857)
0620 02:26:29.329599 (+ 10178us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.329616 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.331524 (+ 1908us) server_negotiation.cc:407] Connection header received
0620 02:26:29.331695 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.331709 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.332066 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.332348 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:29.333830 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.335660 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.339679 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.341563 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.341697 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.341958 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.342887 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.343162 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9735,"thread_start_us":10099,"threads_started":1}
I20260620 02:26:29.348453 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.318847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:38239 (local address 127.0.0.1:33608)
0620 02:26:29.335439 (+ 16592us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:29.335463 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:29.337233 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:29.339780 (+ 2547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:29.339789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:29.339837 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:29.340454 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.340471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.342757 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.342766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.345982 (+ 3216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.345999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.347468 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.347475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.347687 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.347721 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:26:29.347855 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 02:26:29.348010 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16424,"mutex_wait_us":65,"thread_start_us":16189,"threads_started":1}
I20260620 02:26:29.348709 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.319225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.9.231.65:38239)
0620 02:26:29.334617 (+ 15392us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.334629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.335623 (+ 994us) server_negotiation.cc:407] Connection header received
0620 02:26:29.337536 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.337547 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.339010 (+ 1463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.339344 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:29.340707 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.342587 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.346192 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.347228 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.347387 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.347633 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.347909 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.348070 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15173,"thread_start_us":15401,"threads_started":1}
I20260620 02:26:29.349042 29960 tablet_service.cc:1467] Processing CreateTablet for tablet f6c8c15524a5426c980e70c395db9e8d (DEFAULT_TABLE table=test-table [id=4913c911367143d8b91e41d79f663919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:29.350801 29960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c8c15524a5426c980e70c395db9e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:29.351689 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.336132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:33741 (local address 127.0.0.1:47870)
0620 02:26:29.341318 (+ 5186us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:29.341338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:29.341473 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:29.342271 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:29.342276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:29.342309 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:29.342845 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.342858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.345272 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.345279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.348215 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.348232 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.350682 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.350690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.350900 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.350941 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:26:29.351053 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:26:29.351237 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5050,"mutex_wait_us":6,"thread_start_us":4835,"threads_started":1}
I20260620 02:26:29.351718 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.331810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.9.231.66:33741)
0620 02:26:29.340964 (+ 9154us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.340977 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.341607 (+ 630us) server_negotiation.cc:407] Connection header received
0620 02:26:29.341753 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.341762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.342043 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.343354 (+ 1311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:29.343618 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.345088 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.348565 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.350119 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.350293 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.350525 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.351149 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.351382 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8955,"thread_start_us":8778,"threads_started":1}
I20260620 02:26:29.353086 29692 tablet_service.cc:1467] Processing CreateTablet for tablet f6c8c15524a5426c980e70c395db9e8d (DEFAULT_TABLE table=test-table [id=4913c911367143d8b91e41d79f663919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:29.354760 29692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c8c15524a5426c980e70c395db9e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:29.356129 29826 tablet_service.cc:1467] Processing CreateTablet for tablet f6c8c15524a5426c980e70c395db9e8d (DEFAULT_TABLE table=test-table [id=4913c911367143d8b91e41d79f663919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:29.357663 29826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c8c15524a5426c980e70c395db9e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:29.364674 30044 tablet_bootstrap.cc:492] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Bootstrap starting.
I20260620 02:26:29.369583 30044 tablet_bootstrap.cc:654] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:29.371318 30045 tablet_bootstrap.cc:492] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Bootstrap starting.
I20260620 02:26:29.371376 30044 log.cc:824] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:29.375272 30044 tablet_bootstrap.cc:492] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: No bootstrap required, opened a new log
I20260620 02:26:29.375645 30044 ts_tablet_manager.cc:1397] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260620 02:26:29.372979 30047 tablet_bootstrap.cc:492] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Bootstrap starting.
I20260620 02:26:29.376554 30045 tablet_bootstrap.cc:654] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:29.377941 30045 log.cc:824] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:29.379436 30047 tablet_bootstrap.cc:654] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:29.380550 30047 log.cc:824] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:29.386842 30047 tablet_bootstrap.cc:492] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: No bootstrap required, opened a new log
I20260620 02:26:29.387236 30047 ts_tablet_manager.cc:1397] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260620 02:26:29.388826 30045 tablet_bootstrap.cc:492] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: No bootstrap required, opened a new log
I20260620 02:26:29.388731 30044 raft_consensus.cc:348] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.389204 30045 ts_tablet_manager.cc:1397] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.009s
I20260620 02:26:29.389235 30044 raft_consensus.cc:374] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:29.389384 30044 raft_consensus.cc:729] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce065722a94bc297cdbb80c2c0c8d5, State: Initialized, Role: FOLLOWER
I20260620 02:26:29.390092 30044 consensus_queue.cc:260] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.397287 30025 heartbeater.cc:503] Master 127.9.231.126:33907 was elected leader, sending a full tablet report...
I20260620 02:26:29.398154 30044 ts_tablet_manager.cc:1428] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s
I20260620 02:26:29.398120 30045 raft_consensus.cc:348] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.398696 30045 raft_consensus.cc:374] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:29.398885 30045 raft_consensus.cc:729] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43724228200446d8a8a6b704b8bcf3f, State: Initialized, Role: FOLLOWER
I20260620 02:26:29.399637 30045 consensus_queue.cc:260] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.400360 30047 raft_consensus.cc:348] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.401033 30047 raft_consensus.cc:374] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:29.401192 30047 raft_consensus.cc:729] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580f759dd1b241c68070a907ab919ba6, State: Initialized, Role: FOLLOWER
I20260620 02:26:29.401626 30045 ts_tablet_manager.cc:1428] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 02:26:29.402091 30047 consensus_queue.cc:260] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.403671 29891 heartbeater.cc:503] Master 127.9.231.126:33907 was elected leader, sending a full tablet report...
I20260620 02:26:29.404170 30047 ts_tablet_manager.cc:1428] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260620 02:26:29.434557 29892 tablet.cc:2369] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:29.450497 30050 raft_consensus.cc:482] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:26:29.450871 30050 raft_consensus.cc:504] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.452845 30050 leader_election.cc:288] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d43724228200446d8a8a6b704b8bcf3f (127.9.231.65:38239), 580f759dd1b241c68070a907ab919ba6 (127.9.231.66:33741)
I20260620 02:26:29.460536 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.453553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:52297 (local address 127.9.231.66:33741)
0620 02:26:29.453807 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.453816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.453841 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:26:29.453935 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.453942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.454070 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.454182 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:29.454637 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.455709 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.458256 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.459541 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.459721 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.459810 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.460145 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.460278 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:26:29.460601 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.453143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:33741 (local address 127.9.231.67:52297)
0620 02:26:29.453469 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:29.453491 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:29.453647 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:29.454227 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:29.454234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:29.454269 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:29.454494 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.454503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.455863 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.455867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.458042 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.458061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.459732 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.459737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.459898 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.459938 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:26:29.460080 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 02:26:29.460248 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":31}
I20260620 02:26:29.460884 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.453366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:58569 (local address 127.9.231.65:38239)
0620 02:26:29.453585 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.453593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.454301 (+ 708us) server_negotiation.cc:407] Connection header received
0620 02:26:29.454443 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.454447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.454602 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.454709 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:29.455354 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.456250 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.458690 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.459933 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.460093 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.460163 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.460478 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.460628 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:26:29.461050 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.453302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:38239 (local address 127.9.231.67:58569)
0620 02:26:29.454170 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:29.454187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:29.454297 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:29.454776 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:29.454780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:29.454803 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:29.455179 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.455190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.456400 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.456406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.458446 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:29.458456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.460145 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.460151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:29.460307 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.460336 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:26:29.460438 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:26:29.460561 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":62,"thread_start_us":525,"threads_started":1}
I20260620 02:26:29.462054 29846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c8c15524a5426c980e70c395db9e8d" candidate_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580f759dd1b241c68070a907ab919ba6" is_pre_election: true
I20260620 02:26:29.462338 29712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c8c15524a5426c980e70c395db9e8d" candidate_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43724228200446d8a8a6b704b8bcf3f" is_pre_election: true
I20260620 02:26:29.462597 29846 raft_consensus.cc:2454] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce065722a94bc297cdbb80c2c0c8d5 in term 0.
I20260620 02:26:29.462885 29712 raft_consensus.cc:2454] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce065722a94bc297cdbb80c2c0c8d5 in term 0.
I20260620 02:26:29.463446 29914 leader_election.cc:302] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580f759dd1b241c68070a907ab919ba6, 73ce065722a94bc297cdbb80c2c0c8d5; no voters:
I20260620 02:26:29.463896 30050 raft_consensus.cc:2781] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:26:29.464048 30050 raft_consensus.cc:482] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:26:29.464154 30050 raft_consensus.cc:3037] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:29.466596 30050 raft_consensus.cc:504] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.467362 30050 leader_election.cc:288] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [CANDIDATE]: Term 1 election: Requested vote from peers d43724228200446d8a8a6b704b8bcf3f (127.9.231.65:38239), 580f759dd1b241c68070a907ab919ba6 (127.9.231.66:33741)
I20260620 02:26:29.467998 29846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c8c15524a5426c980e70c395db9e8d" candidate_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580f759dd1b241c68070a907ab919ba6"
I20260620 02:26:29.468019 29712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c8c15524a5426c980e70c395db9e8d" candidate_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43724228200446d8a8a6b704b8bcf3f"
I20260620 02:26:29.468252 29846 raft_consensus.cc:3037] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:29.468274 29712 raft_consensus.cc:3037] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:29.471143 29846 raft_consensus.cc:2454] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce065722a94bc297cdbb80c2c0c8d5 in term 1.
I20260620 02:26:29.471273 29712 raft_consensus.cc:2454] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce065722a94bc297cdbb80c2c0c8d5 in term 1.
I20260620 02:26:29.471701 29914 leader_election.cc:302] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580f759dd1b241c68070a907ab919ba6, 73ce065722a94bc297cdbb80c2c0c8d5; no voters:
I20260620 02:26:29.472090 30050 raft_consensus.cc:2781] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:29.472568 30050 raft_consensus.cc:686] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 1 LEADER]: Becoming Leader. State: Replica: 73ce065722a94bc297cdbb80c2c0c8d5, State: Running, Role: LEADER
I20260620 02:26:29.473140 30050 consensus_queue.cc:237] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } }
I20260620 02:26:29.478736 29568 catalog_manager.cc:5495] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ce065722a94bc297cdbb80c2c0c8d5 (127.9.231.67). New cstate: current_term: 1 leader_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73ce065722a94bc297cdbb80c2c0c8d5" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 } health_report { overall_health: UNKNOWN } } }
W20260620 02:26:29.486160 30026 tablet.cc:2369] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:29.513860 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.508161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.9.231.67:37857)
0620 02:26:29.508354 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.508358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.508374 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:26:29.508465 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.508469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.508560 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.508706 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:26:29.509158 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.509915 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.511560 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.511958 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.512100 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.512676 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:26:29.513269 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:26:29.513390 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.513569 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.513646 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:26:29.522653 29712 raft_consensus.cc:1261] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 1 FOLLOWER]: Refusing update from remote peer 73ce065722a94bc297cdbb80c2c0c8d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:29.522660 29846 raft_consensus.cc:1261] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 1 FOLLOWER]: Refusing update from remote peer 73ce065722a94bc297cdbb80c2c0c8d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:29.523818 30054 consensus_queue.cc:1035] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43724228200446d8a8a6b704b8bcf3f" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:29.524261 30050 consensus_queue.cc:1035] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580f759dd1b241c68070a907ab919ba6" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:29.553952 30059 mvcc.cc:205] Tried to move back new op lower bound from 7298754107473649664 to 7298754107286745088. Current Snapshot: MvccSnapshot[applied={T|T < 7298754107473649664}]
I20260620 02:26:29.554175 29568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:29.554330 30058 mvcc.cc:205] Tried to move back new op lower bound from 7298754107473649664 to 7298754107286745088. Current Snapshot: MvccSnapshot[applied={T|T < 7298754107473649664 or (T in {7298754107473649664})}]
I20260620 02:26:29.554522 29568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:29.558668 29568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b80f1140e2c341ffb8dc7d5cd2370d69: Sending DeleteTablet for 3 replicas of tablet f6c8c15524a5426c980e70c395db9e8d
I20260620 02:26:29.560041 29692 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c8c15524a5426c980e70c395db9e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:29 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:33608
I20260620 02:26:29.560141 29960 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c8c15524a5426c980e70c395db9e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:29 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:35842
I20260620 02:26:29.560231 29826 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c8c15524a5426c980e70c395db9e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:29 UTC) from {username='kudu', principal='kudu/127.9.231.126@KRBTEST.COM'} at 127.0.0.1:47870
I20260620 02:26:29.562177 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 29629
I20260620 02:26:29.574249 30069 tablet_replica.cc:330] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: stopping tablet replica
I20260620 02:26:29.574608 30067 tablet_replica.cc:330] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: stopping tablet replica
I20260620 02:26:29.574890 30069 raft_consensus.cc:2229] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:29.574007 30068 tablet_replica.cc:330] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: stopping tablet replica
I20260620 02:26:29.575546 30069 raft_consensus.cc:2258] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:29.575882 30067 raft_consensus.cc:2229] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:26:29.575572 30068 raft_consensus.cc:2229] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:29.576725 30067 raft_consensus.cc:2258] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:29.577316 30068 raft_consensus.cc:2258] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:29.578239 30069 ts_tablet_manager.cc:1905] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:29.579982 30068 ts_tablet_manager.cc:1905] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:29.581557 30067 ts_tablet_manager.cc:1905] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:29.585695 30069 ts_tablet_manager.cc:1918] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 20 02:26:29 dist-test-slave-9vg1 krb5kdc[29523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.65@KRBTEST.COM
I20260620 02:26:29.585963 30069 log.cc:1196] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/f6c8c15524a5426c980e70c395db9e8d
I20260620 02:26:29.586658 30069 ts_tablet_manager.cc:1939] T f6c8c15524a5426c980e70c395db9e8d P 580f759dd1b241c68070a907ab919ba6: Deleting consensus metadata
I20260620 02:26:29.589720 29553 catalog_manager.cc:4841] TS 580f759dd1b241c68070a907ab919ba6 (127.9.231.66:33741): tablet f6c8c15524a5426c980e70c395db9e8d (table test-table [id=4913c911367143d8b91e41d79f663919]) successfully deleted
I20260620 02:26:29.592788 30068 ts_tablet_manager.cc:1918] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:29.593075 30068 log.cc:1196] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-0/wal/wals/f6c8c15524a5426c980e70c395db9e8d
I20260620 02:26:29.593720 30068 ts_tablet_manager.cc:1939] T f6c8c15524a5426c980e70c395db9e8d P d43724228200446d8a8a6b704b8bcf3f: Deleting consensus metadata
I20260620 02:26:29.595321 30067 ts_tablet_manager.cc:1918] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:29.595567 30067 log.cc:1196] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/f6c8c15524a5426c980e70c395db9e8d
I20260620 02:26:29.596007 29554 catalog_manager.cc:4841] TS d43724228200446d8a8a6b704b8bcf3f (127.9.231.65:38239): tablet f6c8c15524a5426c980e70c395db9e8d (table test-table [id=4913c911367143d8b91e41d79f663919]) successfully deleted
I20260620 02:26:29.596222 30067 ts_tablet_manager.cc:1939] T f6c8c15524a5426c980e70c395db9e8d P 73ce065722a94bc297cdbb80c2c0c8d5: Deleting consensus metadata
I20260620 02:26:29.596271 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.563396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.9.231.65:38239)
0620 02:26:29.563604 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.563611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.570766 (+ 7155us) server_negotiation.cc:407] Connection header received
0620 02:26:29.571017 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.571024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.571167 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.571287 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:29.576333 (+ 5046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.577549 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.580370 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.581017 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.581133 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.587206 (+ 6073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:29.587229 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:29.587249 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:29.587318 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:29.592134 (+ 4816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.593286 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.593301 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.593305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.593518 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.594057 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.594062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.594066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.594578 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:29.594905 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.595559 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.595842 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:26:29.597972 29752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:29.598001 29553 catalog_manager.cc:4841] TS 73ce065722a94bc297cdbb80c2c0c8d5 (127.9.231.67:37857): tablet f6c8c15524a5426c980e70c395db9e8d (table test-table [id=4913c911367143d8b91e41d79f663919]) successfully deleted
I20260620 02:26:29.732226 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 29629
I20260620 02:26:29.757726 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 29763
Jun 20 02:26:29 dist-test-slave-9vg1 krb5kdc[29523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.66@KRBTEST.COM
I20260620 02:26:29.776763 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.758721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.9.231.66:33741)
0620 02:26:29.758917 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.758922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.758941 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:26:29.759237 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.759242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.759312 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.759409 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:29.761125 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.761912 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.763997 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.764406 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.764507 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.770196 (+ 5689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:29.770217 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:29.770234 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:29.770278 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:29.773571 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.774452 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.774464 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.774469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.774618 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.775015 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.775023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.775027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.775506 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:29.775833 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.776161 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.776404 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:26:29.777484 29886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:29.906708 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 29763
I20260620 02:26:29.925082 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 29897
Jun 20 02:26:29 dist-test-slave-9vg1 krb5kdc[29523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.67@KRBTEST.COM
I20260620 02:26:29.946455 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:29.925953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.9.231.67:37857)
0620 02:26:29.926133 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:29.926139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:29.926210 (+ 71us) server_negotiation.cc:407] Connection header received
0620 02:26:29.926525 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:29.926530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:29.926613 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:29.926708 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:29.928765 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.929692 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.931657 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:29.932113 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:29.932212 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:29.938229 (+ 6017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:29.938254 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:29.938281 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:29.938342 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:29.942761 (+ 4419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.943895 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.943907 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.943911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.944123 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:29.944547 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:29.944552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:29.944556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:29.945079 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:29.945425 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:29.945785 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 02:26:29.946030 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:26:29.947327 30020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:30.076254 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 29897
I20260620 02:26:30.098500 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 29536
I20260620 02:26:30.099766 29598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:30.213546 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 29536
2026-06-20T02:26:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4738 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T02:26:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:30Z Disabled control of system clock
I20260620 02:26:30.299310 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40077
--webserver_interface=127.9.231.126
--webserver_port=0
--builtin_ntp_servers=127.9.231.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:26:30.653888 30078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:30.654206 30078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:30.654348 30078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:30.662587 30078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:30.662726 30078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:30.662801 30078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:30.662894 30078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:30.662956 30078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:30.672931 30078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:40077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:30.674348 30078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:30.676735 30078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:30.690208 30084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:30.690267 30083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:30.690176 30086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:30.691043 30078 server_base.cc:1034] running on GCE node
I20260620 02:26:30.691956 30078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:30.693778 30078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:30.695039 30078 hybrid_clock.cc:648] HybridClock initialized: now 1781922390694970 us; error 61 us; skew 500 ppm
I20260620 02:26:30.695498 30078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:30.702996 30078 webserver.cc:460] Webserver started at http://127.9.231.126:36715/ using document root <none> and password file <none>
I20260620 02:26:30.703842 30078 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:30.703989 30078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:30.704327 30078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:30.707535 30078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "4ddae10051c7446a9c6a1be32d61f358"
format_stamp: "Formatted at 2026-06-20 02:26:30 on dist-test-slave-9vg1"
server_key: "58a7d6daad9ef26c0c383a19bc4d1a86"
server_key_iv: "bb7b1f0b2722370548cd43c1f9a1ecb5"
server_key_version: "encryptionkey@0"
I20260620 02:26:30.708438 30078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "4ddae10051c7446a9c6a1be32d61f358"
format_stamp: "Formatted at 2026-06-20 02:26:30 on dist-test-slave-9vg1"
server_key: "58a7d6daad9ef26c0c383a19bc4d1a86"
server_key_iv: "bb7b1f0b2722370548cd43c1f9a1ecb5"
server_key_version: "encryptionkey@0"
I20260620 02:26:30.714041 30078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:26:30.717800 30092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:30.719311 30078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:30.719516 30078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "4ddae10051c7446a9c6a1be32d61f358"
format_stamp: "Formatted at 2026-06-20 02:26:30 on dist-test-slave-9vg1"
server_key: "58a7d6daad9ef26c0c383a19bc4d1a86"
server_key_iv: "bb7b1f0b2722370548cd43c1f9a1ecb5"
server_key_version: "encryptionkey@0"
I20260620 02:26:30.719820 30078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:30.735472 30078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:30.736644 30078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:30.736953 30078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:30.757052 30078 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:40077
I20260620 02:26:30.757151 30143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:40077 every 8 connection(s)
I20260620 02:26:30.758526 30078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:30.762684 30144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:30.766911 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30078
I20260620 02:26:30.767140 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:30.767640 10141 external_mini_cluster.cc:1428] Setting key 728dfcf087b4d84626121033966730ac
I20260620 02:26:30.776688 30144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358: Bootstrap starting.
I20260620 02:26:30.780287 30144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:30.781525 30144 log.cc:824] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:30.782017 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:30.769665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.9.231.126:40077)
0620 02:26:30.770829 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:30.770846 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:30.770990 (+ 144us) server_negotiation.cc:407] Connection header received
0620 02:26:30.772893 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:30.772924 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:30.773459 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:30.773911 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:30.774548 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:30.775921 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:30.777398 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:30.778104 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:30.778315 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:30.779604 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:30.779612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:30.779625 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:30.780186 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:30.780292 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:30.780314 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:30.780459 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:30.780693 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 02:26:30.781019 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":387,"threads_started":1}
I20260620 02:26:30.784600 30144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358: No bootstrap required, opened a new log
I20260620 02:26:30.792351 30144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddae10051c7446a9c6a1be32d61f358" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40077 } }
I20260620 02:26:30.792810 30144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:30.792936 30144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddae10051c7446a9c6a1be32d61f358, State: Initialized, Role: FOLLOWER
I20260620 02:26:30.793574 30144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddae10051c7446a9c6a1be32d61f358" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40077 } }
I20260620 02:26:30.793857 30144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:30.794018 30144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:30.794219 30144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:30.796975 30144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddae10051c7446a9c6a1be32d61f358" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40077 } }
I20260620 02:26:30.797612 30144 leader_election.cc:302] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ddae10051c7446a9c6a1be32d61f358; no voters:
I20260620 02:26:30.798267 30144 leader_election.cc:288] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:30.798413 30149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:30.799191 30149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [term 1 LEADER]: Becoming Leader. State: Replica: 4ddae10051c7446a9c6a1be32d61f358, State: Running, Role: LEADER
I20260620 02:26:30.799772 30149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddae10051c7446a9c6a1be32d61f358" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40077 } }
I20260620 02:26:30.800650 30144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:30.802471 30150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ddae10051c7446a9c6a1be32d61f358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddae10051c7446a9c6a1be32d61f358" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40077 } } }
I20260620 02:26:30.802937 30150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:30.803467 30151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ddae10051c7446a9c6a1be32d61f358. Latest consensus state: current_term: 1 leader_uuid: "4ddae10051c7446a9c6a1be32d61f358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddae10051c7446a9c6a1be32d61f358" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 40077 } } }
I20260620 02:26:30.804032 30151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:30.805455 30154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:30.810992 30154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:30.820827 30154 catalog_manager.cc:1347] Generated new cluster ID: 4ae94bb8b55b4fd889ef2315cc011396
I20260620 02:26:30.821026 30154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:30.840018 30154 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:30.841408 30154 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:30.852409 30154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358: Generated new TSK 0
I20260620 02:26:30.853403 30154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:30.860636 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.65:0
--local_ip_for_outbound_sockets=127.9.231.65
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40077
--builtin_ntp_servers=127.9.231.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:26:31.248162 30168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:31.248471 30168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:31.248631 30168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:31.256269 30168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:31.256392 30168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:31.256578 30168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.65
I20260620 02:26:31.266245 30168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.231.65
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.65
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:31.267730 30168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:31.269697 30168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:31.284950 30173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:31.284976 30174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:31.285445 30176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:31.289328 30168 server_base.cc:1034] running on GCE node
I20260620 02:26:31.290086 30168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:31.291337 30168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:31.292629 30168 hybrid_clock.cc:648] HybridClock initialized: now 1781922391292552 us; error 44 us; skew 500 ppm
I20260620 02:26:31.293094 30168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:31.295846 30168 webserver.cc:460] Webserver started at http://127.9.231.65:34147/ using document root <none> and password file <none>
I20260620 02:26:31.296685 30168 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:31.296787 30168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:31.297227 30168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:31.300701 30168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data/instance:
uuid: "a8c14013cee840b688edd61c316d113b"
format_stamp: "Formatted at 2026-06-20 02:26:31 on dist-test-slave-9vg1"
server_key: "b869c085330aed134e92db8bf11d2170"
server_key_iv: "b0c8c8b1841f628ca7da2355fa7c802b"
server_key_version: "encryptionkey@0"
I20260620 02:26:31.301676 30168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance:
uuid: "a8c14013cee840b688edd61c316d113b"
format_stamp: "Formatted at 2026-06-20 02:26:31 on dist-test-slave-9vg1"
server_key: "b869c085330aed134e92db8bf11d2170"
server_key_iv: "b0c8c8b1841f628ca7da2355fa7c802b"
server_key_version: "encryptionkey@0"
I20260620 02:26:31.308413 30168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:26:31.312537 30182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:31.314256 30168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:26:31.314563 30168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/wal
uuid: "a8c14013cee840b688edd61c316d113b"
format_stamp: "Formatted at 2026-06-20 02:26:31 on dist-test-slave-9vg1"
server_key: "b869c085330aed134e92db8bf11d2170"
server_key_iv: "b0c8c8b1841f628ca7da2355fa7c802b"
server_key_version: "encryptionkey@0"
I20260620 02:26:31.314829 30168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:31.335034 30168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:31.336180 30168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:31.336548 30168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:31.337776 30168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:31.339655 30168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:31.339761 30168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:31.339936 30168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:31.340044 30168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:31.396919 30168 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.65:45371
I20260620 02:26:31.397028 30294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.65:45371 every 8 connection(s)
I20260620 02:26:31.398555 30168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/data/info.pb
I20260620 02:26:31.408828 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30168
I20260620 02:26:31.409035 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-0/wal/instance
I20260620 02:26:31.409611 10141 external_mini_cluster.cc:1428] Setting key 9243eaaf1920c73964b8f1a1db370b5a
I20260620 02:26:31.412503 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.66:0
--local_ip_for_outbound_sockets=127.9.231.66
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40077
--builtin_ntp_servers=127.9.231.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:26:31.417240 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:31.401758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:53595 (local address 127.9.231.126:40077)
0620 02:26:31.402279 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:31.402307 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:31.403942 (+ 1635us) server_negotiation.cc:407] Connection header received
0620 02:26:31.406918 (+ 2976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:31.406923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:31.407005 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:31.407456 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:31.409059 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.410196 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.413211 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.413985 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.414151 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:31.416087 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:31.416095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:31.416103 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:31.416141 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:31.416211 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:31.416231 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:31.416329 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:31.416652 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 02:26:31.416782 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":250,"threads_started":1}
I20260620 02:26:31.418701 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:31.402504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40077 (local address 127.9.231.65:53595)
0620 02:26:31.403722 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:31.403783 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:31.406537 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:31.407751 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:31.407764 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:31.407895 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:31.408785 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:31.408809 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.410384 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.410393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:31.413004 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:31.413027 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.414277 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.414286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:31.414681 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:31.415642 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:31.415648 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:31.415715 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:31.415842 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:31.416350 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:31.416366 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:31.416376 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:26:31.416593 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 02:26:31.417103 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":302,"threads_started":1}
I20260620 02:26:31.420833 30295 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40077
I20260620 02:26:31.421222 30295 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:31.422271 30295 heartbeater.cc:511] Master 127.9.231.126:40077 requested a full tablet report, sending...
I20260620 02:26:31.425259 30109 ts_manager.cc:194] Registered new tserver with Master: a8c14013cee840b688edd61c316d113b (127.9.231.65:45371)
I20260620 02:26:31.427665 30109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.65:53595
W20260620 02:26:31.769714 30299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:31.770048 30299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:31.770217 30299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:31.778157 30299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:31.778330 30299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:31.778509 30299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.66
I20260620 02:26:31.788460 30299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.231.66
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.66
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:31.789872 30299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:31.791817 30299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:31.806591 30304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:31.806664 30305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:31.806641 30307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:31.807335 30299 server_base.cc:1034] running on GCE node
I20260620 02:26:31.808177 30299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:31.809317 30299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:31.810639 30299 hybrid_clock.cc:648] HybridClock initialized: now 1781922391810517 us; error 57 us; skew 500 ppm
I20260620 02:26:31.811002 30299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:31.813768 30299 webserver.cc:460] Webserver started at http://127.9.231.66:41927/ using document root <none> and password file <none>
I20260620 02:26:31.814576 30299 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:31.814726 30299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:31.815129 30299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:31.818796 30299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data/instance:
uuid: "182a1b0bf72340fdae4925f74df6b088"
format_stamp: "Formatted at 2026-06-20 02:26:31 on dist-test-slave-9vg1"
server_key: "4b774d91c0a827c60356c425cdc72bd8"
server_key_iv: "b99798097728d19f6b64094c1e24de2d"
server_key_version: "encryptionkey@0"
I20260620 02:26:31.819790 30299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance:
uuid: "182a1b0bf72340fdae4925f74df6b088"
format_stamp: "Formatted at 2026-06-20 02:26:31 on dist-test-slave-9vg1"
server_key: "4b774d91c0a827c60356c425cdc72bd8"
server_key_iv: "b99798097728d19f6b64094c1e24de2d"
server_key_version: "encryptionkey@0"
I20260620 02:26:31.825662 30299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:26:31.829412 30313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:31.830899 30299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:26:31.831151 30299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal
uuid: "182a1b0bf72340fdae4925f74df6b088"
format_stamp: "Formatted at 2026-06-20 02:26:31 on dist-test-slave-9vg1"
server_key: "4b774d91c0a827c60356c425cdc72bd8"
server_key_iv: "b99798097728d19f6b64094c1e24de2d"
server_key_version: "encryptionkey@0"
I20260620 02:26:31.831393 30299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:31.849666 30299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:31.850600 30299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:31.850947 30299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:31.852118 30299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:31.854051 30299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:31.854171 30299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:31.854302 30299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:31.854388 30299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:31.890551 30299 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.66:39745
I20260620 02:26:31.890645 30425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.66:39745 every 8 connection(s)
I20260620 02:26:31.892032 30299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/data/info.pb
I20260620 02:26:31.894904 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30299
I20260620 02:26:31.895133 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal/instance
I20260620 02:26:31.895591 10141 external_mini_cluster.cc:1428] Setting key 615d67bbea820dec297cee0fe7ed01f2
I20260620 02:26:31.898526 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.231.67:0
--local_ip_for_outbound_sockets=127.9.231.67
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40077
--builtin_ntp_servers=127.9.231.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:26:31.906852 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:31.894894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.66:45473 (local address 127.9.231.126:40077)
0620 02:26:31.895190 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:31.895197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:31.896963 (+ 1766us) server_negotiation.cc:407] Connection header received
0620 02:26:31.898991 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:31.898999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:31.899125 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:31.899238 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:31.900373 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.901582 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.903472 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.904220 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.904362 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:31.905862 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:31.905871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:31.905878 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:31.905906 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:31.905951 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:31.905969 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:31.906067 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:31.906322 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 02:26:31.906455 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:26:31.908308 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:31.895536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40077 (local address 127.9.231.66:45473)
0620 02:26:31.896783 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:31.896838 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:31.898643 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:31.899461 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:31.899470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:31.899564 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:31.900197 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:31.900216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.901794 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.901798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:31.903321 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:31.903335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:31.904463 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:31.904477 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:31.904699 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:31.905518 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:31.905521 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:31.905564 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:31.905653 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:31.906083 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:31.906092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:31.906098 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:26:31.906276 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 02:26:31.906615 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":394,"threads_started":1}
I20260620 02:26:31.909870 30426 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40077
I20260620 02:26:31.910200 30426 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:31.910976 30426 heartbeater.cc:511] Master 127.9.231.126:40077 requested a full tablet report, sending...
I20260620 02:26:31.912618 30109 ts_manager.cc:194] Registered new tserver with Master: 182a1b0bf72340fdae4925f74df6b088 (127.9.231.66:39745)
I20260620 02:26:31.913450 30109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.66:45473
W20260620 02:26:32.256121 30429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:32.256512 30429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:32.256678 30429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:32.264317 30429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:32.264501 30429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:32.264685 30429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.231.67
I20260620 02:26:32.274559 30429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.231.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.231.67
--webserver_port=0
--tserver_master_addrs=127.9.231.126:40077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.231.67
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:32.276074 30429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:32.277977 30429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:32.292604 30437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:32.292925 30434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:32.292893 30435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:32.294067 30429 server_base.cc:1034] running on GCE node
I20260620 02:26:32.294799 30429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:32.295959 30429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:32.297254 30429 hybrid_clock.cc:648] HybridClock initialized: now 1781922392297197 us; error 48 us; skew 500 ppm
I20260620 02:26:32.297737 30429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:26:32.300550 30429 webserver.cc:460] Webserver started at http://127.9.231.67:42629/ using document root <none> and password file <none>
I20260620 02:26:32.301414 30429 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:32.301566 30429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:32.301971 30429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:32.305176 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data/instance:
uuid: "93ab65b832f84415904015463ad0b1ac"
format_stamp: "Formatted at 2026-06-20 02:26:32 on dist-test-slave-9vg1"
server_key: "172b50831a529694f806721e4c0c52f9"
server_key_iv: "d6d9fabcf2f28b0aad54e5ede1afac57"
server_key_version: "encryptionkey@0"
I20260620 02:26:32.306054 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance:
uuid: "93ab65b832f84415904015463ad0b1ac"
format_stamp: "Formatted at 2026-06-20 02:26:32 on dist-test-slave-9vg1"
server_key: "172b50831a529694f806721e4c0c52f9"
server_key_iv: "d6d9fabcf2f28b0aad54e5ede1afac57"
server_key_version: "encryptionkey@0"
I20260620 02:26:32.312526 30429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:26:32.316634 30443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:32.318270 30429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:26:32.318499 30429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal
uuid: "93ab65b832f84415904015463ad0b1ac"
format_stamp: "Formatted at 2026-06-20 02:26:32 on dist-test-slave-9vg1"
server_key: "172b50831a529694f806721e4c0c52f9"
server_key_iv: "d6d9fabcf2f28b0aad54e5ede1afac57"
server_key_version: "encryptionkey@0"
I20260620 02:26:32.318861 30429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:32.343003 30429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:32.344214 30429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:32.344600 30429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:32.345799 30429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:26:32.348343 30429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:26:32.348488 30429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:32.348618 30429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:26:32.348752 30429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 02:26:32.385661 30429 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.67:41789
I20260620 02:26:32.385712 30555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.67:41789 every 8 connection(s)
I20260620 02:26:32.387179 30429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/data/info.pb
I20260620 02:26:32.390024 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30429
I20260620 02:26:32.390231 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal/instance
I20260620 02:26:32.390762 10141 external_mini_cluster.cc:1428] Setting key 3d017aa93078bcbed22c5834662678d3
I20260620 02:26:32.402920 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.390564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.67:51571 (local address 127.9.231.126:40077)
0620 02:26:32.390810 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.390818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.392718 (+ 1900us) server_negotiation.cc:407] Connection header received
0620 02:26:32.394787 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.394792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.394862 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.394953 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:32.396500 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.397266 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.399781 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.400192 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.400310 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.402038 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:32.402043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:32.402046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:32.402065 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:32.402101 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:32.402113 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:32.402195 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.402489 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.402589 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:26:32.404170 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.391326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:40077 (local address 127.9.231.67:51571)
0620 02:26:32.392541 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:32.392596 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:32.394503 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:32.395260 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:32.395270 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:32.395366 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:26:32.396177 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.396206 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.397461 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.397467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.399589 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.399608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.400462 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.400470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.400836 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.401607 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:26:32.401611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:26:32.401670 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:26:32.401791 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:26:32.402229 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:26:32.402240 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:26:32.402249 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:26:32.402464 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 02:26:32.402902 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":414,"threads_started":1}
I20260620 02:26:32.405762 30556 heartbeater.cc:347] Connected to a master server at 127.9.231.126:40077
I20260620 02:26:32.406134 30556 heartbeater.cc:464] Registering TS with master...
I20260620 02:26:32.406985 30556 heartbeater.cc:511] Master 127.9.231.126:40077 requested a full tablet report, sending...
I20260620 02:26:32.408610 30109 ts_manager.cc:194] Registered new tserver with Master: 93ab65b832f84415904015463ad0b1ac (127.9.231.67:41789)
I20260620 02:26:32.409693 30109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.231.67:51571
I20260620 02:26:32.419225 10141 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:26:32.430610 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.423824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.9.231.126:40077)
0620 02:26:32.424101 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.424112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.424690 (+ 578us) server_negotiation.cc:407] Connection header received
0620 02:26:32.424822 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.424826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.424895 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.424989 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:32.425606 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.426366 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.427890 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.428440 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.428543 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.429733 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:32.429737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:32.429741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:32.429762 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:32.429801 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:32.429817 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:32.429894 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.430136 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.430231 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:26:32.432303 30295 heartbeater.cc:503] Master 127.9.231.126:40077 was elected leader, sending a full tablet report...
I20260620 02:26:32.435528 30109 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:26:32.443270 30109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:26:32.470975 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.457281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:39745 (local address 127.0.0.1:34826)
0620 02:26:32.459351 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:32.459409 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:32.459657 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:32.463331 (+ 3674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:32.463339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:32.463384 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:32.463862 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.463879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.466088 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.466094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.468194 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.468216 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.469989 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.469998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.470201 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.470271 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 02:26:32.470384 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:26:32.470602 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"thread_start_us":5306,"threads_started":1}
I20260620 02:26:32.471719 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.457326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.9.231.66:39745)
0620 02:26:32.462505 (+ 5179us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.462517 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.462555 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:26:32.462684 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.462693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.463002 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.464065 (+ 1063us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:32.464343 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.465922 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.468447 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.469749 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.470694 (+ 945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.470929 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.471022 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.471307 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5030,"thread_start_us":4912,"threads_started":1}
I20260620 02:26:32.471738 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.457376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50110)
0620 02:26:32.462888 (+ 5512us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:32.462909 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:32.463041 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:32.464049 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:32.464053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:32.464078 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:32.464431 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.464444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.466350 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.466358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.468442 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.468457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.470778 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.470785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.470996 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.471167 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 02:26:32.471262 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:26:32.471400 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5377,"thread_start_us":1382,"threads_started":1}
I20260620 02:26:32.472532 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.457947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.9.231.65:45371)
0620 02:26:32.460451 (+ 2504us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.460465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.463153 (+ 2688us) server_negotiation.cc:407] Connection header received
0620 02:26:32.463419 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.463442 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.463822 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.464034 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:32.464664 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.466186 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.468667 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.470169 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.470311 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.470639 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.472033 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.472230 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2263,"thread_start_us":2082,"threads_started":1}
I20260620 02:26:32.472908 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.461323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:41789 (local address 127.0.0.1:49842)
0620 02:26:32.464035 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:32.464065 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:32.464199 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:32.465245 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:32.465249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:32.465276 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:32.465734 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.465749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.467772 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.467778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.469924 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.469940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.472175 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.472182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.472354 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.472386 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:26:32.472482 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 02:26:32.472609 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2577,"thread_start_us":2406,"threads_started":1}
I20260620 02:26:32.473133 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.462089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.9.231.67:41789)
0620 02:26:32.463473 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.463486 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.464323 (+ 837us) server_negotiation.cc:407] Connection header received
0620 02:26:32.464435 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.464442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.464741 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.464933 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:32.465937 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.467602 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.470198 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.471669 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.471821 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.472114 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.472564 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.472755 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":1006,"threads_started":1}
I20260620 02:26:32.476234 30230 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6b14e4211d480f900f97c02d8a3832 (DEFAULT_TABLE table=test-table [id=6fccd78219a846e4af2e1e0a8726251f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:32.476212 30361 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6b14e4211d480f900f97c02d8a3832 (DEFAULT_TABLE table=test-table [id=6fccd78219a846e4af2e1e0a8726251f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:32.476526 30491 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6b14e4211d480f900f97c02d8a3832 (DEFAULT_TABLE table=test-table [id=6fccd78219a846e4af2e1e0a8726251f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:26:32.477769 30230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6b14e4211d480f900f97c02d8a3832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:32.477789 30361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6b14e4211d480f900f97c02d8a3832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:32.477912 30491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6b14e4211d480f900f97c02d8a3832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:32.490948 30574 tablet_bootstrap.cc:492] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Bootstrap starting.
I20260620 02:26:32.493782 30576 tablet_bootstrap.cc:492] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: Bootstrap starting.
I20260620 02:26:32.494481 30575 tablet_bootstrap.cc:492] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Bootstrap starting.
I20260620 02:26:32.495133 30574 tablet_bootstrap.cc:654] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:32.496537 30574 log.cc:824] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:32.497639 30575 tablet_bootstrap.cc:654] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:32.497962 30576 tablet_bootstrap.cc:654] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:32.498962 30575 log.cc:824] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:32.499250 30576 log.cc:824] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:32.500455 30574 tablet_bootstrap.cc:492] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: No bootstrap required, opened a new log
I20260620 02:26:32.500797 30574 ts_tablet_manager.cc:1397] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 02:26:32.502048 30575 tablet_bootstrap.cc:492] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: No bootstrap required, opened a new log
I20260620 02:26:32.502353 30575 ts_tablet_manager.cc:1397] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 02:26:32.503420 30576 tablet_bootstrap.cc:492] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: No bootstrap required, opened a new log
I20260620 02:26:32.503701 30576 ts_tablet_manager.cc:1397] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 02:26:32.510878 30575 raft_consensus.cc:348] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.511441 30575 raft_consensus.cc:374] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:32.511606 30575 raft_consensus.cc:729] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182a1b0bf72340fdae4925f74df6b088, State: Initialized, Role: FOLLOWER
I20260620 02:26:32.512322 30575 consensus_queue.cc:260] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.512399 30574 raft_consensus.cc:348] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.513056 30574 raft_consensus.cc:374] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:32.513211 30574 raft_consensus.cc:729] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ab65b832f84415904015463ad0b1ac, State: Initialized, Role: FOLLOWER
I20260620 02:26:32.513581 30426 heartbeater.cc:503] Master 127.9.231.126:40077 was elected leader, sending a full tablet report...
I20260620 02:26:32.514079 30574 consensus_queue.cc:260] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.514356 30575 ts_tablet_manager.cc:1428] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260620 02:26:32.515647 30556 heartbeater.cc:503] Master 127.9.231.126:40077 was elected leader, sending a full tablet report...
I20260620 02:26:32.515599 30576 raft_consensus.cc:348] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.516139 30576 raft_consensus.cc:374] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:32.516249 30576 raft_consensus.cc:729] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c14013cee840b688edd61c316d113b, State: Initialized, Role: FOLLOWER
I20260620 02:26:32.516583 30574 ts_tablet_manager.cc:1428] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260620 02:26:32.516935 30576 consensus_queue.cc:260] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.518981 30576 ts_tablet_manager.cc:1428] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260620 02:26:32.639659 30557 tablet.cc:2369] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:26:32.644272 30427 tablet.cc:2369] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:26:32.652055 30296 tablet.cc:2369] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:26:32.703346 30582 raft_consensus.cc:482] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:26:32.703719 30582 raft_consensus.cc:504] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.705528 30582 leader_election.cc:288] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182a1b0bf72340fdae4925f74df6b088 (127.9.231.66:39745), 93ab65b832f84415904015463ad0b1ac (127.9.231.67:41789)
I20260620 02:26:32.713162 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.706078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:57645 (local address 127.9.231.67:41789)
0620 02:26:32.706321 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.706328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.707140 (+ 812us) server_negotiation.cc:407] Connection header received
0620 02:26:32.707401 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.707411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.707472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.707567 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:32.708356 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.709170 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.711340 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.712159 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.712349 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.712423 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.712755 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.712890 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:26:32.713176 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.705963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.231.65:41037 (local address 127.9.231.66:39745)
0620 02:26:32.706190 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.706198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.707251 (+ 1053us) server_negotiation.cc:407] Connection header received
0620 02:26:32.707455 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.707462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.707542 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.707650 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:26:32.708419 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.709085 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.711337 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.712120 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.712310 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.712389 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.712748 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.712892 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:26:32.713296 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.706024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.66:39745 (local address 127.9.231.65:41037)
0620 02:26:32.707147 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:32.707162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:32.707316 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:32.707655 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:32.707659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:32.707729 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:32.708232 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.708242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.709318 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.709323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.711115 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.711141 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.712352 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.712359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.712562 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.712588 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:26:32.712707 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 02:26:32.712887 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":166,"thread_start_us":520,"threads_started":1}
I20260620 02:26:32.713343 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.706020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.67:41789 (local address 127.9.231.65:57645)
0620 02:26:32.706980 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:32.707016 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:26:32.707246 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:26:32.707609 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:26:32.707613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:26:32.707637 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:26:32.708190 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.708211 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.709351 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.709355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.711155 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:26:32.711168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.712390 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.712397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:26:32.712583 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.712606 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:26:32.712718 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:26:32.712883 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":140,"spinlock_wait_cycles":18816,"thread_start_us":488,"threads_started":1}
I20260620 02:26:32.714350 30511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6b14e4211d480f900f97c02d8a3832" candidate_uuid: "a8c14013cee840b688edd61c316d113b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ab65b832f84415904015463ad0b1ac" is_pre_election: true
I20260620 02:26:32.714350 30381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6b14e4211d480f900f97c02d8a3832" candidate_uuid: "a8c14013cee840b688edd61c316d113b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182a1b0bf72340fdae4925f74df6b088" is_pre_election: true
I20260620 02:26:32.714885 30381 raft_consensus.cc:2454] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c14013cee840b688edd61c316d113b in term 0.
I20260620 02:26:32.714886 30511 raft_consensus.cc:2454] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c14013cee840b688edd61c316d113b in term 0.
I20260620 02:26:32.715683 30183 leader_election.cc:302] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182a1b0bf72340fdae4925f74df6b088, a8c14013cee840b688edd61c316d113b; no voters:
I20260620 02:26:32.716126 30582 raft_consensus.cc:2781] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:26:32.716288 30582 raft_consensus.cc:482] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:26:32.716456 30582 raft_consensus.cc:3037] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:32.718791 30582 raft_consensus.cc:504] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.719583 30582 leader_election.cc:288] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [CANDIDATE]: Term 1 election: Requested vote from peers 182a1b0bf72340fdae4925f74df6b088 (127.9.231.66:39745), 93ab65b832f84415904015463ad0b1ac (127.9.231.67:41789)
I20260620 02:26:32.720249 30511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6b14e4211d480f900f97c02d8a3832" candidate_uuid: "a8c14013cee840b688edd61c316d113b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ab65b832f84415904015463ad0b1ac"
I20260620 02:26:32.720229 30381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6b14e4211d480f900f97c02d8a3832" candidate_uuid: "a8c14013cee840b688edd61c316d113b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182a1b0bf72340fdae4925f74df6b088"
I20260620 02:26:32.720541 30511 raft_consensus.cc:3037] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:32.720568 30381 raft_consensus.cc:3037] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:32.723023 30511 raft_consensus.cc:2454] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c14013cee840b688edd61c316d113b in term 1.
I20260620 02:26:32.723102 30381 raft_consensus.cc:2454] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c14013cee840b688edd61c316d113b in term 1.
I20260620 02:26:32.723628 30183 leader_election.cc:302] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 182a1b0bf72340fdae4925f74df6b088, 93ab65b832f84415904015463ad0b1ac, a8c14013cee840b688edd61c316d113b; no voters:
I20260620 02:26:32.724084 30582 raft_consensus.cc:2781] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:32.724623 30582 raft_consensus.cc:686] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 1 LEADER]: Becoming Leader. State: Replica: a8c14013cee840b688edd61c316d113b, State: Running, Role: LEADER
I20260620 02:26:32.725168 30582 consensus_queue.cc:237] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } }
I20260620 02:26:32.730279 30107 catalog_manager.cc:5495] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c14013cee840b688edd61c316d113b (127.9.231.65). New cstate: current_term: 1 leader_uuid: "a8c14013cee840b688edd61c316d113b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c14013cee840b688edd61c316d113b" member_type: VOTER last_known_addr { host: "127.9.231.65" port: 45371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 } health_report { overall_health: UNKNOWN } } }
I20260620 02:26:32.772058 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.764518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.9.231.65:45371)
0620 02:26:32.764746 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.764752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.764803 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:26:32.765043 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.765049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.765185 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.765355 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:26:32.765761 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.766864 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.768815 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.769426 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.769553 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.770295 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:26:32.770999 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:26:32.771223 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.771698 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.771790 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 02:26:32.785584 30511 raft_consensus.cc:1261] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 1 FOLLOWER]: Refusing update from remote peer a8c14013cee840b688edd61c316d113b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:32.785584 30381 raft_consensus.cc:1261] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 1 FOLLOWER]: Refusing update from remote peer a8c14013cee840b688edd61c316d113b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:26:32.786672 30585 consensus_queue.cc:1035] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [LEADER]: Connected to new peer: Peer: permanent_uuid: "182a1b0bf72340fdae4925f74df6b088" member_type: VOTER last_known_addr { host: "127.9.231.66" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:32.787103 30582 consensus_queue.cc:1035] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ab65b832f84415904015463ad0b1ac" member_type: VOTER last_known_addr { host: "127.9.231.67" port: 41789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:26:32.807924 30590 mvcc.cc:205] Tried to move back new op lower bound from 7298754120838967296 to 7298754120606228480. Current Snapshot: MvccSnapshot[applied={T|T < 7298754120838967296}]
I20260620 02:26:32.816992 30109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:32.817337 30109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:26:32.821792 30109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ddae10051c7446a9c6a1be32d61f358: Sending DeleteTablet for 3 replicas of tablet 3a6b14e4211d480f900f97c02d8a3832
I20260620 02:26:32.823231 30361 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6b14e4211d480f900f97c02d8a3832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:32 UTC) from {username='slave'} at 127.0.0.1:34826
I20260620 02:26:32.823717 30230 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6b14e4211d480f900f97c02d8a3832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:32 UTC) from {username='slave'} at 127.0.0.1:50110
I20260620 02:26:32.824122 30491 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6b14e4211d480f900f97c02d8a3832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:26:32 UTC) from {username='slave'} at 127.0.0.1:49842
I20260620 02:26:32.824440 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30168
I20260620 02:26:32.824461 30598 tablet_replica.cc:330] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: stopping tablet replica
I20260620 02:26:32.825067 30598 raft_consensus.cc:2229] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:32.825762 30598 raft_consensus.cc:2258] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:32.828765 30598 ts_tablet_manager.cc:1905] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:32.835865 30600 tablet_replica.cc:330] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: stopping tablet replica
I20260620 02:26:32.836735 30600 raft_consensus.cc:2229] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:26:32.837316 30598 ts_tablet_manager.cc:1918] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:32.837388 30600 raft_consensus.cc:2258] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:32.837523 30598 log.cc:1196] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-1/wal/wals/3a6b14e4211d480f900f97c02d8a3832
I20260620 02:26:32.838119 30598 ts_tablet_manager.cc:1939] T 3a6b14e4211d480f900f97c02d8a3832 P 182a1b0bf72340fdae4925f74df6b088: Deleting consensus metadata
I20260620 02:26:32.840406 30093 catalog_manager.cc:4841] TS 182a1b0bf72340fdae4925f74df6b088 (127.9.231.66:39745): tablet 3a6b14e4211d480f900f97c02d8a3832 (table test-table [id=6fccd78219a846e4af2e1e0a8726251f]) successfully deleted
I20260620 02:26:32.840939 30600 ts_tablet_manager.cc:1905] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:32.843725 30599 tablet_replica.cc:330] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: stopping tablet replica
I20260620 02:26:32.844306 30599 raft_consensus.cc:2229] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:26:32.844380 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.825936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.9.231.65:45371)
0620 02:26:32.826228 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.826236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.834530 (+ 8294us) server_negotiation.cc:407] Connection header received
0620 02:26:32.834944 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.834952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.835188 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.835314 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:32.836093 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.837338 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.839967 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.840650 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.840769 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.842422 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:32.842433 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:32.842451 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:32.843325 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:32.843408 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:32.843434 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:32.843528 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.843816 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.843960 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:26:32.845386 30599 raft_consensus.cc:2258] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:26:32.845794 30290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:32.849699 30600 ts_tablet_manager.cc:1918] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:26:32.850011 30600 log.cc:1196] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Deleting WAL directory at /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922055280103-10141-0/minicluster-data/ts-2/wal/wals/3a6b14e4211d480f900f97c02d8a3832
I20260620 02:26:32.850641 30600 ts_tablet_manager.cc:1939] T 3a6b14e4211d480f900f97c02d8a3832 P 93ab65b832f84415904015463ad0b1ac: Deleting consensus metadata
I20260620 02:26:32.852540 30095 catalog_manager.cc:4841] TS 93ab65b832f84415904015463ad0b1ac (127.9.231.67:41789): tablet 3a6b14e4211d480f900f97c02d8a3832 (table test-table [id=6fccd78219a846e4af2e1e0a8726251f]) successfully deleted
I20260620 02:26:32.951740 30599 ts_tablet_manager.cc:1905] T 3a6b14e4211d480f900f97c02d8a3832 P a8c14013cee840b688edd61c316d113b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:26:32.952164 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30168
W20260620 02:26:32.972586 30095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.231.65:45371 (error 108)
I20260620 02:26:32.973371 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30299
I20260620 02:26:32.975142 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.974270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50152)
0620 02:26:32.974781 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:32.974901 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":324,"threads_started":1}
W20260620 02:26:32.975395 30095 catalog_manager.cc:4584] TS a8c14013cee840b688edd61c316d113b (127.9.231.65:45371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a6b14e4211d480f900f97c02d8a3832: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
I20260620 02:26:32.981371 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:32.974478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.9.231.66:39745)
0620 02:26:32.974665 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:32.974671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:32.974695 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:26:32.974768 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:32.974773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:32.974880 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:32.975005 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:32.975550 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.976365 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.977725 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:32.978166 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:32.978332 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:32.979813 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:32.979826 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:32.979842 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:32.980508 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:32.980568 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:32.980596 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:32.980734 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:32.980886 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:26:32.981019 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:26:32.982162 30421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:33.029268 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:33.028750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50166)
0620 02:26:33.029004 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:33.029101 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 02:26:33.030699 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:33.030222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50178)
0620 02:26:33.030403 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:33.030531 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:26:33.075317 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30299
I20260620 02:26:33.091353 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:33.090722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50182)
0620 02:26:33.090990 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:33.091134 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:26:33.092769 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:33.092227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50188)
0620 02:26:33.092436 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:33.092553 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:26:33.094584 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30429
I20260620 02:26:33.102208 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:33.095439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.9.231.67:41789)
0620 02:26:33.095618 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:33.095624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:33.095654 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:26:33.095805 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:33.095809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:33.095913 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:33.096002 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:33.096409 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:33.097216 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:33.098747 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:33.099307 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:33.099414 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:33.100716 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:33.100726 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:33.100741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:33.101425 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:33.101500 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:33.101532 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:33.101649 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:33.101799 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 02:26:33.101903 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:26:33.102900 30551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:33.173786 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:33.173149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50204)
0620 02:26:33.173454 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:33.173574 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 02:26:33.175050 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:33.174509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.65:45371 (local address 127.0.0.1:50216)
0620 02:26:33.174754 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 02:26:33.174842 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.65:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 02:26:33.196913 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30429
I20260620 02:26:33.217260 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30078
I20260620 02:26:33.218568 30139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:33.341907 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30078
2026-06-20T02:26:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3131 ms)
[----------] 6 tests from EncryptionPolicyTest (23357 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:33 dist-test-slave-9vg1 krb5kdc[30610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:33 dist-test-slave-9vg1 krb5kdc[30610](info): set up 2 sockets
Jun 20 02:26:33 dist-test-slave-9vg1 krb5kdc[30610](info): commencing operation
krb5kdc: starting...
W20260620 02:26:34.489149 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:34 dist-test-slave-9vg1 krb5kdc[30610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:34.775933 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:35.156554 30623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:35.156977 30623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:35.157166 30623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:35.165771 30623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:35.165992 30623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:35.166121 30623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:35.166231 30623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:35.166320 30623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:35.176776 30623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:35.178274 30623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:35.179768 30623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:35.192920 30628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:35.193100 30631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:35.193121 30629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:35.194262 30623 server_base.cc:1034] running on GCE node
I20260620 02:26:35.195456 30623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:35.197135 30623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:35.198359 30623 hybrid_clock.cc:648] HybridClock initialized: now 1781922395198311 us; error 52 us; skew 500 ppm
I20260620 02:26:35.198777 30623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:35 dist-test-slave-9vg1 krb5kdc[30610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:35.202483 30623 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:35.204370 30623 webserver.cc:460] Webserver started at http://127.0.0.1:37417/ using document root <none> and password file <none>
I20260620 02:26:35.205116 30623 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:35.205260 30623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:35.205610 30623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:35.209007 30623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "a34022e54ba3499aa1c12350f9d15c9b"
format_stamp: "Formatted at 2026-06-20 02:26:35 on dist-test-slave-9vg1"
server_key: "b6890a82f737d385182fd9bb53b8391e"
server_key_iv: "103401f5d8c323d04da95a549c3d0979"
server_key_version: "encryptionkey@0"
I20260620 02:26:35.209889 30623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "a34022e54ba3499aa1c12350f9d15c9b"
format_stamp: "Formatted at 2026-06-20 02:26:35 on dist-test-slave-9vg1"
server_key: "b6890a82f737d385182fd9bb53b8391e"
server_key_iv: "103401f5d8c323d04da95a549c3d0979"
server_key_version: "encryptionkey@0"
I20260620 02:26:35.215763 30623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:26:35.220033 30638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:35.221598 30623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:26:35.221866 30623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "a34022e54ba3499aa1c12350f9d15c9b"
format_stamp: "Formatted at 2026-06-20 02:26:35 on dist-test-slave-9vg1"
server_key: "b6890a82f737d385182fd9bb53b8391e"
server_key_iv: "103401f5d8c323d04da95a549c3d0979"
server_key_version: "encryptionkey@0"
I20260620 02:26:35.222100 30623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:35.249711 30623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:35.254781 30623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:35.255189 30623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:35.277343 30623 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42293
I20260620 02:26:35.277395 30689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42293 every 8 connection(s)
I20260620 02:26:35.279096 30623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:35.283658 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30623
I20260620 02:26:35.283987 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:35.284123 30690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:35.284904 10141 external_mini_cluster.cc:1428] Setting key 9ca320a8dd1df9af3205f39179921334
I20260620 02:26:35.302547 30690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b: Bootstrap starting.
Jun 20 02:26:35 dist-test-slave-9vg1 krb5kdc[30610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:26:35.307479 30690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:35.309098 30690 log.cc:824] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:35.313073 30690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b: No bootstrap required, opened a new log
I20260620 02:26:35.314863 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:35.287404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40901 (local address 127.0.0.1:42293)
0620 02:26:35.288367 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:35.288379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:35.288430 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:26:35.290217 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:35.290245 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:35.290770 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:35.291273 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:35.293500 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:35.294959 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:35.297240 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:35.298148 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:35.298358 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:35.304602 (+ 6244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:35.304619 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:35.304651 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:35.304725 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:35.309310 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:35.310631 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:35.310640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:35.310685 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:35.310929 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:35.311504 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:35.311510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:35.311513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:35.312279 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:35.312650 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:35.313051 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 02:26:35.313415 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":458,"threads_started":1}
I20260620 02:26:35.319620 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:35.323532 30690 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34022e54ba3499aa1c12350f9d15c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42293 } }
I20260620 02:26:35.324035 30690 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:35.324215 30690 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34022e54ba3499aa1c12350f9d15c9b, State: Initialized, Role: FOLLOWER
I20260620 02:26:35.325003 30690 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34022e54ba3499aa1c12350f9d15c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42293 } }
I20260620 02:26:35.325306 30690 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:35.325491 30690 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:35.325783 30690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:35.329255 30690 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34022e54ba3499aa1c12350f9d15c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42293 } }
I20260620 02:26:35.329941 30690 leader_election.cc:302] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34022e54ba3499aa1c12350f9d15c9b; no voters:
I20260620 02:26:35.330801 30690 leader_election.cc:288] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:35.331050 30700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:35.331840 30700 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [term 1 LEADER]: Becoming Leader. State: Replica: a34022e54ba3499aa1c12350f9d15c9b, State: Running, Role: LEADER
I20260620 02:26:35.332711 30700 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34022e54ba3499aa1c12350f9d15c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42293 } }
I20260620 02:26:35.334697 30690 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:35.334817 30700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34022e54ba3499aa1c12350f9d15c9b. Latest consensus state: current_term: 1 leader_uuid: "a34022e54ba3499aa1c12350f9d15c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34022e54ba3499aa1c12350f9d15c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42293 } } }
I20260620 02:26:35.335229 30700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:35.335783 30702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a34022e54ba3499aa1c12350f9d15c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34022e54ba3499aa1c12350f9d15c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42293 } } }
I20260620 02:26:35.336449 30702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:35.342262 30704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:35.342361 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:35.324681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44921 (local address 127.0.0.1:42293)
0620 02:26:35.324952 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:35.324962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:35.325542 (+ 580us) server_negotiation.cc:407] Connection header received
0620 02:26:35.325751 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:35.325758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:35.325893 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:35.326043 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:35.327835 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:35.329043 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:35.330865 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:35.331622 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:35.331792 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:35.335088 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:35.335098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:35.335104 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:35.335156 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:35.338472 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:35.339608 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:35.339614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:35.339617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:35.339778 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:35.340417 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:35.340423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:35.340426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:35.340996 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:35.341315 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:35.341739 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 02:26:35.341948 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:26:35.350853 30704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:35.362006 30704 catalog_manager.cc:1347] Generated new cluster ID: 196a8c7882834d9f90bf78be40c4c104
I20260620 02:26:35.362156 30704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:35.383929 30704 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:35.386013 30704 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:35.397744 30704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a34022e54ba3499aa1c12350f9d15c9b: Generated new TSK 0
I20260620 02:26:35.398746 30704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:35.412456 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30623
I20260620 02:26:35.414273 30685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:35.514489 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30623
W20260620 02:26:35.533291 30608 connection.cc:535] client connection to 127.0.0.1:42293 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42293): Connection reset by peer (error 104)
2026-06-20T02:26:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2161 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:35 dist-test-slave-9vg1 krb5kdc[30722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:35 dist-test-slave-9vg1 krb5kdc[30722](info): set up 2 sockets
Jun 20 02:26:35 dist-test-slave-9vg1 krb5kdc[30722](info): commencing operation
krb5kdc: starting...
W20260620 02:26:36.672693 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:36 dist-test-slave-9vg1 krb5kdc[30722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:36.959010 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:37.307400 30735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:37.307723 30735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:37.307819 30735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:37.315307 30735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:37.315434 30735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:37.315487 30735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:37.315542 30735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:37.315579 30735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:37.326582 30735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:37.328056 30735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:37.329888 30735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:37.343487 30743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:37.343461 30740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:37.343955 30741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:37.344439 30735 server_base.cc:1034] running on GCE node
I20260620 02:26:37.345448 30735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:37.347162 30735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:37.348543 30735 hybrid_clock.cc:648] HybridClock initialized: now 1781922397348473 us; error 57 us; skew 500 ppm
I20260620 02:26:37.348997 30735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:37 dist-test-slave-9vg1 krb5kdc[30722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:37.358286 30735 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:37.360174 30735 webserver.cc:460] Webserver started at http://127.0.0.1:41773/ using document root <none> and password file <none>
I20260620 02:26:37.360937 30735 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:37.361097 30735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:37.361426 30735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:37.364719 30735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "37eaf040765345529b3ddd39d20a527f"
format_stamp: "Formatted at 2026-06-20 02:26:37 on dist-test-slave-9vg1"
server_key: "df10703eeac49830d6ec105dbd5169ee"
server_key_iv: "c712217c6e3ba1b3fe400ec732afc053"
server_key_version: "encryptionkey@0"
I20260620 02:26:37.365662 30735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "37eaf040765345529b3ddd39d20a527f"
format_stamp: "Formatted at 2026-06-20 02:26:37 on dist-test-slave-9vg1"
server_key: "df10703eeac49830d6ec105dbd5169ee"
server_key_iv: "c712217c6e3ba1b3fe400ec732afc053"
server_key_version: "encryptionkey@0"
I20260620 02:26:37.372982 30735 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 02:26:37.377301 30750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:37.378927 30735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:26:37.379216 30735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "37eaf040765345529b3ddd39d20a527f"
format_stamp: "Formatted at 2026-06-20 02:26:37 on dist-test-slave-9vg1"
server_key: "df10703eeac49830d6ec105dbd5169ee"
server_key_iv: "c712217c6e3ba1b3fe400ec732afc053"
server_key_version: "encryptionkey@0"
I20260620 02:26:37.379529 30735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:37.398788 30735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:37.403829 30735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:37.404155 30735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:37.424213 30735 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38911
I20260620 02:26:37.424266 30801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38911 every 8 connection(s)
I20260620 02:26:37.425758 30735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:37.426906 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30735
I20260620 02:26:37.427166 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:37.427656 10141 external_mini_cluster.cc:1428] Setting key f53a5a14c0eeb21afcc63a77977b43c4
I20260620 02:26:37.430801 30802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:37.444259 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:37.429739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50119 (local address 127.0.0.1:38911)
0620 02:26:37.430588 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:37.430603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:37.430651 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:26:37.433016 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:37.433046 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:37.433556 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:37.434248 (+ 692us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:37.434787 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:37.436959 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:37.438433 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:37.439518 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:37.439719 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:37.441076 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:37.441089 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:37.441117 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:37.441889 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:37.441994 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:37.442035 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:37.442213 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:37.442603 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 02:26:37.442810 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":403,"threads_started":1}
I20260620 02:26:37.445695 30802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f: Bootstrap starting.
I20260620 02:26:37.448745 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:37.449245 30802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:37.450316 30802 log.cc:824] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:37.453787 30802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f: No bootstrap required, opened a new log
I20260620 02:26:37.460845 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:37.453795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36509 (local address 127.0.0.1:38911)
0620 02:26:37.453987 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:37.453994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:37.454347 (+ 353us) server_negotiation.cc:407] Connection header received
0620 02:26:37.454512 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:37.454517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:37.454600 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:37.454724 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:37.455275 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:37.456482 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:37.457834 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:37.458540 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:37.458675 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:37.459984 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:37.459991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:37.459996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:37.460025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:37.460085 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:37.460116 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:37.460243 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:37.460413 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 02:26:37.460475 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:26:37.464051 30802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eaf040765345529b3ddd39d20a527f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38911 } }
I20260620 02:26:37.464468 30802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:37.464551 30802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eaf040765345529b3ddd39d20a527f, State: Initialized, Role: FOLLOWER
I20260620 02:26:37.465138 30802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eaf040765345529b3ddd39d20a527f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38911 } }
I20260620 02:26:37.465348 30802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:37.465447 30802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:37.465588 30802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:37.468125 30802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eaf040765345529b3ddd39d20a527f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38911 } }
I20260620 02:26:37.468645 30802 leader_election.cc:302] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37eaf040765345529b3ddd39d20a527f; no voters:
I20260620 02:26:37.469161 30802 leader_election.cc:288] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:37.469365 30812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:37.470091 30812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [term 1 LEADER]: Becoming Leader. State: Replica: 37eaf040765345529b3ddd39d20a527f, State: Running, Role: LEADER
I20260620 02:26:37.470791 30812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eaf040765345529b3ddd39d20a527f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38911 } }
I20260620 02:26:37.471508 30802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:37.473677 30813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37eaf040765345529b3ddd39d20a527f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eaf040765345529b3ddd39d20a527f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38911 } } }
I20260620 02:26:37.474083 30813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:37.474838 30814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37eaf040765345529b3ddd39d20a527f. Latest consensus state: current_term: 1 leader_uuid: "37eaf040765345529b3ddd39d20a527f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eaf040765345529b3ddd39d20a527f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38911 } } }
I20260620 02:26:37.475287 30814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:37.476351 30820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:37.483129 30820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:37.497117 30820 catalog_manager.cc:1347] Generated new cluster ID: 1fb130f434774ed090b0e2e53bf0ac8a
I20260620 02:26:37.497251 30820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:37.506937 30820 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:37.508294 30820 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:37.515507 30820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37eaf040765345529b3ddd39d20a527f: Generated new TSK 0
I20260620 02:26:37.516610 30820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:37.523849 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30735
W20260620 02:26:37.525091 30797 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50119
W20260620 02:26:37.526005 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30735: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:37.526269 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30735
W20260620 02:26:37.541805 30720 connection.cc:535] client connection to 127.0.0.1:38911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38911): Connection reset by peer (error 104)
2026-06-20T02:26:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:37 dist-test-slave-9vg1 krb5kdc[30834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:37 dist-test-slave-9vg1 krb5kdc[30834](info): set up 2 sockets
Jun 20 02:26:37 dist-test-slave-9vg1 krb5kdc[30834](info): commencing operation
krb5kdc: starting...
W20260620 02:26:38.656672 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:38 dist-test-slave-9vg1 krb5kdc[30834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:38.946744 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:39.309237 30847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:39.309602 30847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:39.309796 30847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:39.317487 30847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:39.317632 30847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:39.317728 30847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:39.317840 30847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:39.317939 30847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:39.328810 30847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:39.330325 30847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:39.331826 30847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:39.345304 30855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:39.345367 30852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:39.345358 30853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:39.346293 30847 server_base.cc:1034] running on GCE node
I20260620 02:26:39.347239 30847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:39.348693 30847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:39.349967 30847 hybrid_clock.cc:648] HybridClock initialized: now 1781922399349906 us; error 29 us; skew 500 ppm
I20260620 02:26:39.350358 30847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:39 dist-test-slave-9vg1 krb5kdc[30834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:39.354218 30847 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:39.356148 30847 webserver.cc:460] Webserver started at http://127.0.0.1:33181/ using document root <none> and password file <none>
I20260620 02:26:39.356897 30847 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:39.357052 30847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:39.357371 30847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:39.360852 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "8e28eddf911b4533a65f3c8de3507698"
format_stamp: "Formatted at 2026-06-20 02:26:39 on dist-test-slave-9vg1"
server_key: "a6d72092a5e93a3f051bfdb5d5322c72"
server_key_iv: "204981eeeb4116ede69b085e799e8b7e"
server_key_version: "encryptionkey@0"
I20260620 02:26:39.361730 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "8e28eddf911b4533a65f3c8de3507698"
format_stamp: "Formatted at 2026-06-20 02:26:39 on dist-test-slave-9vg1"
server_key: "a6d72092a5e93a3f051bfdb5d5322c72"
server_key_iv: "204981eeeb4116ede69b085e799e8b7e"
server_key_version: "encryptionkey@0"
I20260620 02:26:39.367405 30847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:26:39.371193 30862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:39.372606 30847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:26:39.372857 30847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "8e28eddf911b4533a65f3c8de3507698"
format_stamp: "Formatted at 2026-06-20 02:26:39 on dist-test-slave-9vg1"
server_key: "a6d72092a5e93a3f051bfdb5d5322c72"
server_key_iv: "204981eeeb4116ede69b085e799e8b7e"
server_key_version: "encryptionkey@0"
I20260620 02:26:39.373092 30847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:39.380352 30847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:39.385070 30847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:39.385340 30847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:39.405063 30847 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43465
I20260620 02:26:39.405107 30913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43465 every 8 connection(s)
I20260620 02:26:39.406559 30847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:39.410437 30914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:39.414918 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30847
I20260620 02:26:39.415167 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:39.415728 10141 external_mini_cluster.cc:1428] Setting key 8cfd0ab88fc310152f31d79fff180658
I20260620 02:26:39.423926 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:39.417613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46421 (local address 127.0.0.1:43465)
0620 02:26:39.418484 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:39.418496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:39.418541 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:26:39.420217 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:39.420242 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:39.420635 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:39.420991 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:39.422135 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:39.422167 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:39.422181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:39.422197 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:39.422263 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:39.422285 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:39.422478 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:39.422619 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 02:26:39.422858 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":392,"threads_started":1}
I20260620 02:26:39.424727 30914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698: Bootstrap starting.
I20260620 02:26:39.428018 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:39.428474 30914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:39.429610 30914 log.cc:824] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:39.433116 30914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698: No bootstrap required, opened a new log
I20260620 02:26:39.435453 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:39.432750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52199 (local address 127.0.0.1:43465)
0620 02:26:39.432948 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:39.432957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:39.433392 (+ 435us) server_negotiation.cc:407] Connection header received
0620 02:26:39.433612 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:39.433616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:39.433672 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:39.433787 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:39.434735 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:39.434740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:39.434745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:39.434773 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:39.434815 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:39.434836 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:39.434973 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:39.435104 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 02:26:39.435175 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:26:39.441728 30914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e28eddf911b4533a65f3c8de3507698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260620 02:26:39.442175 30914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:39.442291 30914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e28eddf911b4533a65f3c8de3507698, State: Initialized, Role: FOLLOWER
I20260620 02:26:39.442904 30914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e28eddf911b4533a65f3c8de3507698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260620 02:26:39.443271 30914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:39.443435 30914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:39.443635 30914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:39.445894 30914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e28eddf911b4533a65f3c8de3507698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260620 02:26:39.446455 30914 leader_election.cc:302] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e28eddf911b4533a65f3c8de3507698; no voters:
I20260620 02:26:39.447191 30914 leader_election.cc:288] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:39.447365 30924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:39.448117 30924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [term 1 LEADER]: Becoming Leader. State: Replica: 8e28eddf911b4533a65f3c8de3507698, State: Running, Role: LEADER
I20260620 02:26:39.448688 30924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e28eddf911b4533a65f3c8de3507698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260620 02:26:39.449443 30914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:39.451871 30926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e28eddf911b4533a65f3c8de3507698. Latest consensus state: current_term: 1 leader_uuid: "8e28eddf911b4533a65f3c8de3507698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e28eddf911b4533a65f3c8de3507698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } } }
I20260620 02:26:39.451871 30925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e28eddf911b4533a65f3c8de3507698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e28eddf911b4533a65f3c8de3507698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } } }
I20260620 02:26:39.452517 30925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:39.452517 30926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:39.453644 30931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:39.459542 30931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:39.468770 30931 catalog_manager.cc:1347] Generated new cluster ID: 86daa965ec6f4c29b366ebe11b4e93f1
I20260620 02:26:39.468901 30931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:39.481024 30931 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:39.481500 30931 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:39.487072 30931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e28eddf911b4533a65f3c8de3507698: Generated new TSK 0
I20260620 02:26:39.487995 30931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:39.504861 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30847
W20260620 02:26:39.506150 30909 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46421
W20260620 02:26:39.506948 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30847: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:39.507164 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30847
2026-06-20T02:26:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:39 dist-test-slave-9vg1 krb5kdc[30945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:39 dist-test-slave-9vg1 krb5kdc[30945](info): set up 2 sockets
Jun 20 02:26:39 dist-test-slave-9vg1 krb5kdc[30945](info): commencing operation
krb5kdc: starting...
W20260620 02:26:40.636757 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:40 dist-test-slave-9vg1 krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:40.926643 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:41.297173 30958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:41.297575 30958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:41.297755 30958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:41.305653 30958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:41.305842 30958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:41.305967 30958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:41.306083 30958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:41.306165 30958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:41.317087 30958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:41.318583 30958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:41.320070 30958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:41.333101 30963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:41.333197 30966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:41.333880 30964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:41.334477 30958 server_base.cc:1034] running on GCE node
I20260620 02:26:41.335559 30958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:41.337256 30958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:41.338490 30958 hybrid_clock.cc:648] HybridClock initialized: now 1781922401338392 us; error 99 us; skew 500 ppm
I20260620 02:26:41.338892 30958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:41 dist-test-slave-9vg1 krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:41.342664 30958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:41.344699 30958 webserver.cc:460] Webserver started at http://127.0.0.1:40747/ using document root <none> and password file <none>
I20260620 02:26:41.345482 30958 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:41.345643 30958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:41.345990 30958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:41.349442 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "681de86e5c41471b9056337d418a5386"
format_stamp: "Formatted at 2026-06-20 02:26:41 on dist-test-slave-9vg1"
server_key: "e9bb7eac81a057930c934f00a8ec8bff"
server_key_iv: "33f6c782b0618d9f3e6c4c5edfc4fa2d"
server_key_version: "encryptionkey@0"
I20260620 02:26:41.350365 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "681de86e5c41471b9056337d418a5386"
format_stamp: "Formatted at 2026-06-20 02:26:41 on dist-test-slave-9vg1"
server_key: "e9bb7eac81a057930c934f00a8ec8bff"
server_key_iv: "33f6c782b0618d9f3e6c4c5edfc4fa2d"
server_key_version: "encryptionkey@0"
I20260620 02:26:41.356313 30958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:26:41.360318 30973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:41.361873 30958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:26:41.362131 30958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "681de86e5c41471b9056337d418a5386"
format_stamp: "Formatted at 2026-06-20 02:26:41 on dist-test-slave-9vg1"
server_key: "e9bb7eac81a057930c934f00a8ec8bff"
server_key_iv: "33f6c782b0618d9f3e6c4c5edfc4fa2d"
server_key_version: "encryptionkey@0"
I20260620 02:26:41.362377 30958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:41.396353 30958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:41.402128 30958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:41.402483 30958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:41.425494 30958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41039
I20260620 02:26:41.425532 31024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41039 every 8 connection(s)
I20260620 02:26:41.427318 30958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:41.431876 31025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:41.436429 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 30958
I20260620 02:26:41.436591 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:41.437071 10141 external_mini_cluster.cc:1428] Setting key c3915486ab8a7db926b9652a82c6a1d5
I20260620 02:26:41.445159 31025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386: Bootstrap starting.
I20260620 02:26:41.449424 31025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:41.450798 31025 log.cc:824] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:41.454890 31025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386: No bootstrap required, opened a new log
Jun 20 02:26:41 dist-test-slave-9vg1 krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:26:41.463410 31025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681de86e5c41471b9056337d418a5386" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41039 } }
I20260620 02:26:41.463923 31025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:41.464062 31025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681de86e5c41471b9056337d418a5386, State: Initialized, Role: FOLLOWER
I20260620 02:26:41.464707 31025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681de86e5c41471b9056337d418a5386" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41039 } }
I20260620 02:26:41.464980 31025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:41.465158 31025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:41.465349 31025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:41.465437 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:41.439308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54289 (local address 127.0.0.1:41039)
0620 02:26:41.440211 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:41.440223 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:41.440266 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:26:41.443293 (+ 3027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:41.443325 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:41.443805 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:41.444217 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:41.446385 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:41.447960 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:41.449969 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:41.450767 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:41.450933 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:41.456455 (+ 5522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:41.456465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:41.456483 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:41.456543 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:41.460270 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:41.461174 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:41.461182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:41.461190 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:41.461404 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:41.461852 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:41.461863 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:41.461866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:41.462527 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:41.462857 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:41.463367 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 02:26:41.463918 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":429,"threads_started":1}
I20260620 02:26:41.468349 31025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681de86e5c41471b9056337d418a5386" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41039 } }
I20260620 02:26:41.469009 31025 leader_election.cc:302] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 681de86e5c41471b9056337d418a5386; no voters:
I20260620 02:26:41.469882 31025 leader_election.cc:288] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:41.470023 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:41.470088 31030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:41.472467 31025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:41.475414 31030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [term 1 LEADER]: Becoming Leader. State: Replica: 681de86e5c41471b9056337d418a5386, State: Running, Role: LEADER
I20260620 02:26:41.476130 31030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681de86e5c41471b9056337d418a5386" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41039 } }
I20260620 02:26:41.478960 31031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "681de86e5c41471b9056337d418a5386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681de86e5c41471b9056337d418a5386" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41039 } } }
I20260620 02:26:41.479418 31031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:41.480043 31032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681de86e5c41471b9056337d418a5386. Latest consensus state: current_term: 1 leader_uuid: "681de86e5c41471b9056337d418a5386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681de86e5c41471b9056337d418a5386" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41039 } } }
I20260620 02:26:41.480334 31032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:41.482664 31040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:41.490293 31040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:41.514094 31040 catalog_manager.cc:1347] Generated new cluster ID: 565cb2e7c0384b488753d72217f84e34
I20260620 02:26:41.514231 31040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:41.524781 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:41.500712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40317 (local address 127.0.0.1:41039)
0620 02:26:41.500922 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:41.500930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:41.502526 (+ 1596us) server_negotiation.cc:407] Connection header received
0620 02:26:41.502765 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:41.502773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:41.502893 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:41.503016 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:41.505974 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:41.507438 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:41.510128 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:41.511226 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:41.511390 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:41.515976 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:41.515993 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:41.516001 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:41.516073 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:41.521171 (+ 5098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:41.522173 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:41.522179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:41.522183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:41.522366 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:41.522823 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:41.522828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:41.522832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:41.523282 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:41.523516 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:41.524119 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 02:26:41.524374 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 02:26:41.530725 31040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:41.532822 31040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:41.549340 31040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 681de86e5c41471b9056337d418a5386: Generated new TSK 0
I20260620 02:26:41.550398 31040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:41.559202 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 30958
I20260620 02:26:41.560693 31020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:41.657176 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 30958
2026-06-20T02:26:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2151 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:41 dist-test-slave-9vg1 krb5kdc[31058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:41 dist-test-slave-9vg1 krb5kdc[31058](info): set up 2 sockets
Jun 20 02:26:41 dist-test-slave-9vg1 krb5kdc[31058](info): commencing operation
krb5kdc: starting...
W20260620 02:26:42.796715 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:42 dist-test-slave-9vg1 krb5kdc[31058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:43.087478 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:43.459929 31071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:43.460336 31071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:43.460517 31071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:43.468223 31071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:43.468407 31071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:43.468534 31071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:43.468653 31071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:43.468732 31071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:43.479641 31071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:43.481138 31071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:43.482645 31071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:43.496430 31076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:43.496470 31077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:43.496511 31079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:43.497421 31071 server_base.cc:1034] running on GCE node
I20260620 02:26:43.498418 31071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:43.500097 31071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:43.501335 31071 hybrid_clock.cc:648] HybridClock initialized: now 1781922403501279 us; error 34 us; skew 500 ppm
I20260620 02:26:43.501811 31071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:43 dist-test-slave-9vg1 krb5kdc[31058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:43.510697 31071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:43.512655 31071 webserver.cc:460] Webserver started at http://127.0.0.1:42973/ using document root <none> and password file <none>
I20260620 02:26:43.513481 31071 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:43.513607 31071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:43.513957 31071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:43.517470 31071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf"
format_stamp: "Formatted at 2026-06-20 02:26:43 on dist-test-slave-9vg1"
server_key: "2f778f092a1b26afc52ebf62471d3676"
server_key_iv: "f0f91903fc330d633096ca5d8cedc0b2"
server_key_version: "encryptionkey@0"
I20260620 02:26:43.518328 31071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf"
format_stamp: "Formatted at 2026-06-20 02:26:43 on dist-test-slave-9vg1"
server_key: "2f778f092a1b26afc52ebf62471d3676"
server_key_iv: "f0f91903fc330d633096ca5d8cedc0b2"
server_key_version: "encryptionkey@0"
I20260620 02:26:43.524215 31071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:26:43.528266 31086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:43.529893 31071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:26:43.530146 31071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf"
format_stamp: "Formatted at 2026-06-20 02:26:43 on dist-test-slave-9vg1"
server_key: "2f778f092a1b26afc52ebf62471d3676"
server_key_iv: "f0f91903fc330d633096ca5d8cedc0b2"
server_key_version: "encryptionkey@0"
I20260620 02:26:43.530369 31071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:43.550361 31071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:43.554843 31071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:43.555212 31071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:43.575351 31071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35431
I20260620 02:26:43.575389 31137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35431 every 8 connection(s)
I20260620 02:26:43.576855 31071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:43.577349 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31071
I20260620 02:26:43.577585 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:43.578246 10141 external_mini_cluster.cc:1428] Setting key 055da52300310c85ef0495486d371c5c
I20260620 02:26:43.582276 31138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:43.595654 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:43.580410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60973 (local address 127.0.0.1:35431)
0620 02:26:43.581397 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:43.581413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:43.581467 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:26:43.584044 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:43.584074 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:43.584628 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:43.585111 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:43.585852 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:43.587963 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:43.589655 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:43.590759 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:43.590960 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:43.592427 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:43.592439 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:43.592462 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:43.593280 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:43.593389 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:43.593434 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:43.593560 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:43.593810 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 02:26:43.594135 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":448,"threads_started":1}
I20260620 02:26:43.597312 31138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf: Bootstrap starting.
I20260620 02:26:43.600211 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:43.601060 31138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:43.602439 31138 log.cc:824] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:43.606390 31138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf: No bootstrap required, opened a new log
I20260620 02:26:43.613816 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:43.606013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45229 (local address 127.0.0.1:35431)
0620 02:26:43.606265 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:43.606276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:43.606720 (+ 444us) server_negotiation.cc:407] Connection header received
0620 02:26:43.606930 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:43.606934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:43.607033 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:43.607190 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:43.607809 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:43.608697 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:43.610678 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:43.611285 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:43.611393 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:43.612790 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:43.612795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:43.612799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:43.612819 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:43.612865 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:43.612881 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:43.612964 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:43.613151 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 02:26:43.613245 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:26:43.616868 31138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35431 } }
I20260620 02:26:43.617295 31138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:43.617413 31138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc67a2ab2e4498ebea562be3a0ec6cf, State: Initialized, Role: FOLLOWER
I20260620 02:26:43.618081 31138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35431 } }
I20260620 02:26:43.618422 31138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:43.618604 31138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:43.618819 31138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:43.621210 31138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35431 } }
I20260620 02:26:43.621791 31138 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc67a2ab2e4498ebea562be3a0ec6cf; no voters:
I20260620 02:26:43.622515 31138 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:43.622666 31148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:43.623425 31148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [term 1 LEADER]: Becoming Leader. State: Replica: 5cc67a2ab2e4498ebea562be3a0ec6cf, State: Running, Role: LEADER
I20260620 02:26:43.624082 31148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35431 } }
I20260620 02:26:43.624913 31138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:43.627194 31149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35431 } } }
I20260620 02:26:43.627723 31149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:43.627533 31150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc67a2ab2e4498ebea562be3a0ec6cf. Latest consensus state: current_term: 1 leader_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc67a2ab2e4498ebea562be3a0ec6cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35431 } } }
I20260620 02:26:43.627863 31150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:43.630405 31156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:43.636281 31156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:43.646327 31156 catalog_manager.cc:1347] Generated new cluster ID: 2124171e4ebb4ddfa4204a26b97932b0
I20260620 02:26:43.646461 31156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:43.662168 31156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:43.664256 31156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:43.681461 31156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cc67a2ab2e4498ebea562be3a0ec6cf: Generated new TSK 0
I20260620 02:26:43.682729 31156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:43.694005 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31071
W20260620 02:26:43.695396 31133 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60973
W20260620 02:26:43.696192 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31071: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:43.696350 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31071
2026-06-20T02:26:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:43 dist-test-slave-9vg1 krb5kdc[31169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:43 dist-test-slave-9vg1 krb5kdc[31169](info): set up 2 sockets
Jun 20 02:26:43 dist-test-slave-9vg1 krb5kdc[31169](info): commencing operation
krb5kdc: starting...
W20260620 02:26:44.838640 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:45 dist-test-slave-9vg1 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:45.147934 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:45.510428 31182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:45.510767 31182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:45.510931 31182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:45.518496 31182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:45.518628 31182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:45.518676 31182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:45.518781 31182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:45.518873 31182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:45.530282 31182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:45.531953 31182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:45.533529 31182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:45.548486 31187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:45.548521 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:45.548566 31188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:45.549387 31182 server_base.cc:1034] running on GCE node
I20260620 02:26:45.550292 31182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:45.551994 31182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:45.553241 31182 hybrid_clock.cc:648] HybridClock initialized: now 1781922405553193 us; error 45 us; skew 500 ppm
I20260620 02:26:45.553674 31182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:45 dist-test-slave-9vg1 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:45.557318 31182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:45.559145 31182 webserver.cc:460] Webserver started at http://127.0.0.1:37877/ using document root <none> and password file <none>
I20260620 02:26:45.559923 31182 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:45.560045 31182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:45.560328 31182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:45.563711 31182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "3e219bb4045c4dd2b634a5e3f971b975"
format_stamp: "Formatted at 2026-06-20 02:26:45 on dist-test-slave-9vg1"
server_key: "d15e64177363990523a3ff450818476f"
server_key_iv: "b53a9bd2383bfb2b7bb291e89022a65c"
server_key_version: "encryptionkey@0"
I20260620 02:26:45.564563 31182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "3e219bb4045c4dd2b634a5e3f971b975"
format_stamp: "Formatted at 2026-06-20 02:26:45 on dist-test-slave-9vg1"
server_key: "d15e64177363990523a3ff450818476f"
server_key_iv: "b53a9bd2383bfb2b7bb291e89022a65c"
server_key_version: "encryptionkey@0"
I20260620 02:26:45.570550 31182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:26:45.574636 31197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:45.576442 31182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:45.576645 31182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "3e219bb4045c4dd2b634a5e3f971b975"
format_stamp: "Formatted at 2026-06-20 02:26:45 on dist-test-slave-9vg1"
server_key: "d15e64177363990523a3ff450818476f"
server_key_iv: "b53a9bd2383bfb2b7bb291e89022a65c"
server_key_version: "encryptionkey@0"
I20260620 02:26:45.576887 31182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:45.583492 31182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:45.588539 31182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:45.588918 31182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:45.609164 31182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37951
I20260620 02:26:45.609241 31248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37951 every 8 connection(s)
I20260620 02:26:45.610735 31182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:45.614933 31249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:45.616815 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31182
I20260620 02:26:45.617023 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:45.617616 10141 external_mini_cluster.cc:1428] Setting key fb744e3d5949b32f0989d56f22326d45
I20260620 02:26:45.627547 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:45.619992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50939 (local address 127.0.0.1:37951)
0620 02:26:45.621181 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:45.621197 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:45.621254 (+ 57us) server_negotiation.cc:407] Connection header received
0620 02:26:45.623474 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:45.623504 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:45.623929 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:45.624342 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:45.625612 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:45.625623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:45.625640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:45.625658 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:45.625739 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:45.625770 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:45.625998 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:45.626108 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 02:26:45.626363 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":543,"threads_started":1}
I20260620 02:26:45.628504 31249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975: Bootstrap starting.
I20260620 02:26:45.632091 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:45.632640 31249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:45.634023 31249 log.cc:824] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:45.637616 31249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975: No bootstrap required, opened a new log
I20260620 02:26:45.640254 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:45.637620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44971 (local address 127.0.0.1:37951)
0620 02:26:45.637817 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:45.637824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:45.638130 (+ 306us) server_negotiation.cc:407] Connection header received
0620 02:26:45.638275 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:45.638281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:45.638354 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:45.638484 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:45.639512 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:45.639518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:45.639522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:45.639547 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:45.639598 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:45.639621 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:45.639764 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:45.639857 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 02:26:45.639921 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:26:45.646950 31249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e219bb4045c4dd2b634a5e3f971b975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260620 02:26:45.647457 31249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:45.647539 31249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e219bb4045c4dd2b634a5e3f971b975, State: Initialized, Role: FOLLOWER
I20260620 02:26:45.648097 31249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e219bb4045c4dd2b634a5e3f971b975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260620 02:26:45.648317 31249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:45.648425 31249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:45.648559 31249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:45.650873 31249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e219bb4045c4dd2b634a5e3f971b975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260620 02:26:45.651566 31249 leader_election.cc:302] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e219bb4045c4dd2b634a5e3f971b975; no voters:
I20260620 02:26:45.652300 31249 leader_election.cc:288] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:45.652472 31259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:45.653177 31259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [term 1 LEADER]: Becoming Leader. State: Replica: 3e219bb4045c4dd2b634a5e3f971b975, State: Running, Role: LEADER
I20260620 02:26:45.653764 31259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e219bb4045c4dd2b634a5e3f971b975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260620 02:26:45.654623 31249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:45.657006 31260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e219bb4045c4dd2b634a5e3f971b975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e219bb4045c4dd2b634a5e3f971b975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } } }
I20260620 02:26:45.657461 31260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:45.659158 31261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e219bb4045c4dd2b634a5e3f971b975. Latest consensus state: current_term: 1 leader_uuid: "3e219bb4045c4dd2b634a5e3f971b975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e219bb4045c4dd2b634a5e3f971b975" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } } }
I20260620 02:26:45.659713 31261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:45.666399 31266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:45.672163 31266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:45.683329 31266 catalog_manager.cc:1347] Generated new cluster ID: fbcd8d5d9aff473da8dd5303367afb58
I20260620 02:26:45.683473 31266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:45.699817 31266 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:45.700338 31266 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:45.712106 31266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e219bb4045c4dd2b634a5e3f971b975: Generated new TSK 0
I20260620 02:26:45.713050 31266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:45.725572 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31182
W20260620 02:26:45.726958 31244 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50939
W20260620 02:26:45.727803 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31182: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:45.727967 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31182
2026-06-20T02:26:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2032 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:45 dist-test-slave-9vg1 krb5kdc[31280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:45 dist-test-slave-9vg1 krb5kdc[31280](info): set up 2 sockets
Jun 20 02:26:45 dist-test-slave-9vg1 krb5kdc[31280](info): commencing operation
krb5kdc: starting...
W20260620 02:26:46.889330 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:47 dist-test-slave-9vg1 krb5kdc[31280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:47.176069 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:47.544569 31293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:47.544934 31293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:47.545138 31293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:47.552879 31293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:47.553013 31293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:47.553097 31293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:47.553200 31293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:47.553261 31293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:47.563845 31293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:47.565361 31293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:47.566850 31293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:47.580281 31298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:47.580338 31301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:47.580367 31299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:47.581457 31293 server_base.cc:1034] running on GCE node
I20260620 02:26:47.582348 31293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:47.584072 31293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:47.585306 31293 hybrid_clock.cc:648] HybridClock initialized: now 1781922407585257 us; error 39 us; skew 500 ppm
I20260620 02:26:47.585701 31293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:47 dist-test-slave-9vg1 krb5kdc[31280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:47.589552 31293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:47.591346 31293 webserver.cc:460] Webserver started at http://127.0.0.1:46041/ using document root <none> and password file <none>
I20260620 02:26:47.592113 31293 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:47.592263 31293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:47.592629 31293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:47.596597 31293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "9d139389276a41db8fabc3d967df2544"
format_stamp: "Formatted at 2026-06-20 02:26:47 on dist-test-slave-9vg1"
server_key: "9cab6c572d67e5c5cc38e01346aa7abd"
server_key_iv: "47e24214bfd9f711974b1074cfdc669c"
server_key_version: "encryptionkey@0"
I20260620 02:26:47.597556 31293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "9d139389276a41db8fabc3d967df2544"
format_stamp: "Formatted at 2026-06-20 02:26:47 on dist-test-slave-9vg1"
server_key: "9cab6c572d67e5c5cc38e01346aa7abd"
server_key_iv: "47e24214bfd9f711974b1074cfdc669c"
server_key_version: "encryptionkey@0"
I20260620 02:26:47.603766 31293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 02:26:47.608276 31308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:47.609930 31293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:26:47.610159 31293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "9d139389276a41db8fabc3d967df2544"
format_stamp: "Formatted at 2026-06-20 02:26:47 on dist-test-slave-9vg1"
server_key: "9cab6c572d67e5c5cc38e01346aa7abd"
server_key_iv: "47e24214bfd9f711974b1074cfdc669c"
server_key_version: "encryptionkey@0"
I20260620 02:26:47.610394 31293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:47.629123 31293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:47.633870 31293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:47.634195 31293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:47.654812 31293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39553
I20260620 02:26:47.654912 31359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39553 every 8 connection(s)
I20260620 02:26:47.656452 31293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:47.660853 31360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:47.665483 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31293
I20260620 02:26:47.665704 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:47.666278 10141 external_mini_cluster.cc:1428] Setting key b681467d074dcfefe612ca396c805097
I20260620 02:26:47.675933 31360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544: Bootstrap starting.
I20260620 02:26:47.679541 31360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:47.680759 31360 log.cc:824] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:26:47 dist-test-slave-9vg1 krb5kdc[31280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:26:47.684437 31360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544: No bootstrap required, opened a new log
I20260620 02:26:47.692481 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:47.668417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:43955 (local address 127.0.0.1:39553)
0620 02:26:47.669300 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:47.669313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:47.669357 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:26:47.671205 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:47.671234 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:47.671755 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:47.672146 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:47.674207 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:47.676084 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:47.677661 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:47.678460 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:47.678714 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:47.683816 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:47.683825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:47.683842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:47.683882 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:47.687452 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:47.688501 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:47.688509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:47.688521 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:47.688728 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:47.689247 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:47.689255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:47.689257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:47.689933 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:47.690330 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:47.690850 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 02:26:47.691312 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":401,"threads_started":1}
I20260620 02:26:47.693056 31360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d139389276a41db8fabc3d967df2544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39553 } }
I20260620 02:26:47.693567 31360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:47.693696 31360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d139389276a41db8fabc3d967df2544, State: Initialized, Role: FOLLOWER
I20260620 02:26:47.694355 31360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d139389276a41db8fabc3d967df2544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39553 } }
I20260620 02:26:47.694648 31360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:47.694801 31360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:47.695220 31360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:47.697667 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:47.698287 31360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d139389276a41db8fabc3d967df2544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39553 } }
I20260620 02:26:47.698920 31360 leader_election.cc:302] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d139389276a41db8fabc3d967df2544; no voters:
I20260620 02:26:47.699609 31360 leader_election.cc:288] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:47.699841 31365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:47.701197 31365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [term 1 LEADER]: Becoming Leader. State: Replica: 9d139389276a41db8fabc3d967df2544, State: Running, Role: LEADER
I20260620 02:26:47.702402 31360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:47.702282 31365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d139389276a41db8fabc3d967df2544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39553 } }
I20260620 02:26:47.708428 31366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d139389276a41db8fabc3d967df2544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d139389276a41db8fabc3d967df2544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39553 } } }
I20260620 02:26:47.709034 31366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:47.709750 31367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d139389276a41db8fabc3d967df2544. Latest consensus state: current_term: 1 leader_uuid: "9d139389276a41db8fabc3d967df2544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d139389276a41db8fabc3d967df2544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39553 } } }
I20260620 02:26:47.710193 31367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:47.716070 31376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:47.725216 31376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:47.740366 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:47.708930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:52799 (local address 127.0.0.1:39553)
0620 02:26:47.712923 (+ 3993us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:47.712933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:47.719587 (+ 6654us) server_negotiation.cc:407] Connection header received
0620 02:26:47.719875 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:47.719881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:47.719988 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:47.720099 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:47.722418 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:47.723918 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:47.726271 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:47.727047 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:47.727200 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:47.731275 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:47.731284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:47.731290 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:47.731342 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:47.735365 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:47.736573 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:47.736584 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:47.736588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:47.736752 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:47.737408 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:47.737414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:47.737417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:47.737885 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:47.738192 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:47.739727 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0620 02:26:47.739965 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3762}
I20260620 02:26:47.750639 31376 catalog_manager.cc:1347] Generated new cluster ID: c89b4d2b974f49e080ba181f94eaa394
I20260620 02:26:47.750803 31376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:47.782052 31376 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:47.783612 31376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:47.798290 31376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d139389276a41db8fabc3d967df2544: Generated new TSK 0
I20260620 02:26:47.799463 31376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:47.808897 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31293
I20260620 02:26:47.810212 31355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:47.908349 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31293
W20260620 02:26:47.927018 31278 connection.cc:535] client connection to 127.0.0.1:39553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39553): Connection reset by peer (error 104)
2026-06-20T02:26:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2179 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:48 dist-test-slave-9vg1 krb5kdc[31393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:48 dist-test-slave-9vg1 krb5kdc[31393](info): set up 2 sockets
Jun 20 02:26:48 dist-test-slave-9vg1 krb5kdc[31393](info): commencing operation
krb5kdc: starting...
W20260620 02:26:49.040751 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:49 dist-test-slave-9vg1 krb5kdc[31393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:49.326623 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:49.687913 31406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:49.688242 31406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:49.688338 31406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:49.695806 31406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:49.695937 31406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:49.695986 31406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:49.696049 31406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:49.696094 31406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:49.706456 31406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:49.707902 31406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:49.709666 31406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:49.722033 31411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:49.722066 31414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:49.722369 31412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:49.723049 31406 server_base.cc:1034] running on GCE node
I20260620 02:26:49.724005 31406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:49.725658 31406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:49.726876 31406 hybrid_clock.cc:648] HybridClock initialized: now 1781922409726814 us; error 58 us; skew 500 ppm
I20260620 02:26:49.727288 31406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:49 dist-test-slave-9vg1 krb5kdc[31393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:49.735044 31406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:49.736943 31406 webserver.cc:460] Webserver started at http://127.0.0.1:41823/ using document root <none> and password file <none>
I20260620 02:26:49.737668 31406 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:49.737776 31406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:49.738055 31406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:49.741500 31406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "fff105eb433b45ad9fbd1857a8130f06"
format_stamp: "Formatted at 2026-06-20 02:26:49 on dist-test-slave-9vg1"
server_key: "ebe4345be52b9d47fb354d3e55eecc30"
server_key_iv: "48e6615b22d1bca6a3ee0f318e4ab09a"
server_key_version: "encryptionkey@0"
I20260620 02:26:49.742323 31406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "fff105eb433b45ad9fbd1857a8130f06"
format_stamp: "Formatted at 2026-06-20 02:26:49 on dist-test-slave-9vg1"
server_key: "ebe4345be52b9d47fb354d3e55eecc30"
server_key_iv: "48e6615b22d1bca6a3ee0f318e4ab09a"
server_key_version: "encryptionkey@0"
I20260620 02:26:49.748610 31406 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:26:49.753095 31421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:49.754997 31406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 02:26:49.755316 31406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "fff105eb433b45ad9fbd1857a8130f06"
format_stamp: "Formatted at 2026-06-20 02:26:49 on dist-test-slave-9vg1"
server_key: "ebe4345be52b9d47fb354d3e55eecc30"
server_key_iv: "48e6615b22d1bca6a3ee0f318e4ab09a"
server_key_version: "encryptionkey@0"
I20260620 02:26:49.755563 31406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:49.792351 31406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:49.797204 31406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:49.797569 31406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:49.817747 31406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36115
I20260620 02:26:49.817783 31472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36115 every 8 connection(s)
I20260620 02:26:49.819341 31406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:49.823696 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:49.825567 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31406
I20260620 02:26:49.825788 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:49.826388 10141 external_mini_cluster.cc:1428] Setting key c1ce1e71cf01b76dd11f67147fc4e61a
I20260620 02:26:49.838160 31473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06: Bootstrap starting.
I20260620 02:26:49.840472 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:49.828478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:41865 (local address 127.0.0.1:36115)
0620 02:26:49.829368 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:49.829380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:49.829428 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:26:49.831242 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:49.831275 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:49.831763 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:49.832210 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:49.832804 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:49.834100 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:49.835597 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:49.836358 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:49.836580 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:49.837927 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:49.837935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:49.837951 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:49.838570 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:49.838686 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:49.838714 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:49.838884 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:49.839200 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:26:49.839400 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":391,"threads_started":1}
I20260620 02:26:49.842199 31473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:49.843547 31473 log.cc:824] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:49.845192 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:49.847448 31473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06: No bootstrap required, opened a new log
I20260620 02:26:49.858407 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:49.850856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:49155 (local address 127.0.0.1:36115)
0620 02:26:49.851097 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:49.851104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:49.851685 (+ 581us) server_negotiation.cc:407] Connection header received
0620 02:26:49.851872 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:49.851880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:49.851973 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:49.852110 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:49.852661 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:49.853921 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:49.855516 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:49.856275 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:49.856430 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:49.857549 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:49.857555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:49.857560 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:49.857584 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:49.857656 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:49.857679 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:49.857802 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:49.857985 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 02:26:49.858049 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:26:49.858299 31473 raft_consensus.cc:348] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff105eb433b45ad9fbd1857a8130f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260620 02:26:49.858870 31473 raft_consensus.cc:374] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:49.859004 31473 raft_consensus.cc:729] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff105eb433b45ad9fbd1857a8130f06, State: Initialized, Role: FOLLOWER
I20260620 02:26:49.859788 31473 consensus_queue.cc:260] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff105eb433b45ad9fbd1857a8130f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260620 02:26:49.860155 31473 raft_consensus.cc:388] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:49.860313 31473 raft_consensus.cc:482] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:49.860502 31473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:49.862979 31473 raft_consensus.cc:504] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff105eb433b45ad9fbd1857a8130f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260620 02:26:49.863601 31473 leader_election.cc:302] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fff105eb433b45ad9fbd1857a8130f06; no voters:
I20260620 02:26:49.864295 31473 leader_election.cc:288] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:49.864459 31483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:49.865309 31483 raft_consensus.cc:686] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [term 1 LEADER]: Becoming Leader. State: Replica: fff105eb433b45ad9fbd1857a8130f06, State: Running, Role: LEADER
I20260620 02:26:49.865980 31483 consensus_queue.cc:237] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff105eb433b45ad9fbd1857a8130f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260620 02:26:49.866976 31473 sys_catalog.cc:564] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:49.869063 31484 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fff105eb433b45ad9fbd1857a8130f06. Latest consensus state: current_term: 1 leader_uuid: "fff105eb433b45ad9fbd1857a8130f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff105eb433b45ad9fbd1857a8130f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260620 02:26:49.869601 31484 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:49.870534 31485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fff105eb433b45ad9fbd1857a8130f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff105eb433b45ad9fbd1857a8130f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260620 02:26:49.871026 31485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:49.876624 31489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:49.882789 31489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:49.892944 31489 catalog_manager.cc:1347] Generated new cluster ID: 1ce6f36647c14b9889d7f4d33537d156
I20260620 02:26:49.893138 31489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:49.928524 31489 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:49.930027 31489 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:49.942804 31489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fff105eb433b45ad9fbd1857a8130f06: Generated new TSK 0
I20260620 02:26:49.943784 31489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:49.957767 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31406
W20260620 02:26:49.959219 31468 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.35:41865
W20260620 02:26:49.960045 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:49.960219 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31406
W20260620 02:26:49.978427 31391 connection.cc:535] client connection to 127.0.0.1:36115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36115): Connection reset by peer (error 104)
2026-06-20T02:26:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:50 dist-test-slave-9vg1 krb5kdc[31504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:50 dist-test-slave-9vg1 krb5kdc[31504](info): set up 2 sockets
Jun 20 02:26:50 dist-test-slave-9vg1 krb5kdc[31504](info): commencing operation
krb5kdc: starting...
W20260620 02:26:51.097465 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:51 dist-test-slave-9vg1 krb5kdc[31504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:51.395236 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:51.763172 31517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:51.763496 31517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:51.763602 31517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:51.771024 31517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:51.771195 31517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:51.771243 31517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:51.771299 31517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:51.771344 31517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:51.781769 31517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:51.783277 31517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:51.784607 31517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:51.797758 31523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:51.797861 31525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:51.797778 31522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:51.798698 31517 server_base.cc:1034] running on GCE node
I20260620 02:26:51.799731 31517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:51.801442 31517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:51.802841 31517 hybrid_clock.cc:648] HybridClock initialized: now 1781922411802784 us; error 40 us; skew 500 ppm
I20260620 02:26:51.803336 31517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:51 dist-test-slave-9vg1 krb5kdc[31504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:51.806893 31517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:51.808691 31517 webserver.cc:460] Webserver started at http://127.0.0.1:43037/ using document root <none> and password file <none>
I20260620 02:26:51.809405 31517 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:51.809530 31517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:51.809803 31517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:51.813148 31517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "c23cd5dea2e7446482ba28f80ebe165e"
format_stamp: "Formatted at 2026-06-20 02:26:51 on dist-test-slave-9vg1"
server_key: "8c4ead986f75d2ab150f131054503f66"
server_key_iv: "e715bbc3ae1649b470d0a5ebe7cc1195"
server_key_version: "encryptionkey@0"
I20260620 02:26:51.813973 31517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "c23cd5dea2e7446482ba28f80ebe165e"
format_stamp: "Formatted at 2026-06-20 02:26:51 on dist-test-slave-9vg1"
server_key: "8c4ead986f75d2ab150f131054503f66"
server_key_iv: "e715bbc3ae1649b470d0a5ebe7cc1195"
server_key_version: "encryptionkey@0"
I20260620 02:26:51.819782 31517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:26:51.824079 31532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:51.825820 31517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:26:51.826058 31517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "c23cd5dea2e7446482ba28f80ebe165e"
format_stamp: "Formatted at 2026-06-20 02:26:51 on dist-test-slave-9vg1"
server_key: "8c4ead986f75d2ab150f131054503f66"
server_key_iv: "e715bbc3ae1649b470d0a5ebe7cc1195"
server_key_version: "encryptionkey@0"
I20260620 02:26:51.826280 31517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:51.833062 31517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:51.838043 31517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:51.838392 31517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:51.858999 31517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36415
I20260620 02:26:51.859041 31583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36415 every 8 connection(s)
I20260620 02:26:51.860615 31517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:51.864698 31584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:51.864784 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31517
I20260620 02:26:51.864981 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:51.865445 10141 external_mini_cluster.cc:1428] Setting key a66487b2455ff8813f25393a7e7a154c
I20260620 02:26:51.875164 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:51.867446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:34831 (local address 127.0.0.1:36415)
0620 02:26:51.868363 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:51.868380 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:51.868441 (+ 61us) server_negotiation.cc:407] Connection header received
0620 02:26:51.870346 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:51.870382 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:51.870945 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:51.871607 (+ 662us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:51.872855 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:51.872866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:51.872891 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:51.872910 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:51.872998 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:51.873035 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:51.873337 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:51.873498 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 02:26:51.873746 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":336,"threads_started":1}
I20260620 02:26:51.879467 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:51.881079 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e: Bootstrap starting.
I20260620 02:26:51.885735 31584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:51.887189 31584 log.cc:824] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:51.888447 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:51.885354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:46887 (local address 127.0.0.1:36415)
0620 02:26:51.885568 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:51.885576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:51.886024 (+ 448us) server_negotiation.cc:407] Connection header received
0620 02:26:51.886211 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:51.886219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:51.886308 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:51.886430 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:51.887599 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:51.887607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:51.887613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:51.887639 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:51.887692 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:51.887719 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:51.887888 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:51.888040 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:26:51.888110 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:26:51.890799 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e: No bootstrap required, opened a new log
I20260620 02:26:51.899111 31584 raft_consensus.cc:348] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23cd5dea2e7446482ba28f80ebe165e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260620 02:26:51.899598 31584 raft_consensus.cc:374] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:51.899704 31584 raft_consensus.cc:729] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23cd5dea2e7446482ba28f80ebe165e, State: Initialized, Role: FOLLOWER
I20260620 02:26:51.900372 31584 consensus_queue.cc:260] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23cd5dea2e7446482ba28f80ebe165e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260620 02:26:51.900605 31584 raft_consensus.cc:388] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:51.900779 31584 raft_consensus.cc:482] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:51.900952 31584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:51.903390 31584 raft_consensus.cc:504] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23cd5dea2e7446482ba28f80ebe165e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260620 02:26:51.904057 31584 leader_election.cc:302] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c23cd5dea2e7446482ba28f80ebe165e; no voters:
I20260620 02:26:51.904729 31584 leader_election.cc:288] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:51.904892 31594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:51.905568 31594 raft_consensus.cc:686] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [term 1 LEADER]: Becoming Leader. State: Replica: c23cd5dea2e7446482ba28f80ebe165e, State: Running, Role: LEADER
I20260620 02:26:51.906108 31594 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23cd5dea2e7446482ba28f80ebe165e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260620 02:26:51.906942 31584 sys_catalog.cc:564] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:51.909266 31595 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c23cd5dea2e7446482ba28f80ebe165e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23cd5dea2e7446482ba28f80ebe165e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } } }
I20260620 02:26:51.909235 31596 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c23cd5dea2e7446482ba28f80ebe165e. Latest consensus state: current_term: 1 leader_uuid: "c23cd5dea2e7446482ba28f80ebe165e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23cd5dea2e7446482ba28f80ebe165e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } } }
I20260620 02:26:51.909847 31596 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:51.909843 31595 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:51.915658 31600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:51.921583 31600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:51.931573 31600 catalog_manager.cc:1347] Generated new cluster ID: 1ba55dd9ac1c46a78f378a9c10826418
I20260620 02:26:51.931722 31600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:51.956705 31600 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:51.957177 31600 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:51.966382 31600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c23cd5dea2e7446482ba28f80ebe165e: Generated new TSK 0
I20260620 02:26:51.967401 31600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:51.980978 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31517
W20260620 02:26:51.982260 31579 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.35:34831
W20260620 02:26:51.983304 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31517: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:51.983464 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31517
2026-06-20T02:26:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:52 dist-test-slave-9vg1 krb5kdc[31615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:52 dist-test-slave-9vg1 krb5kdc[31615](info): set up 2 sockets
Jun 20 02:26:52 dist-test-slave-9vg1 krb5kdc[31615](info): commencing operation
krb5kdc: starting...
W20260620 02:26:53.120795 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:53 dist-test-slave-9vg1 krb5kdc[31615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:53.407887 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:53.764431 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:53.764744 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:53.764905 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:53.772485 31628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:53.772630 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:53.772716 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:53.772811 31628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:53.772872 31628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:53.783355 31628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:53.784788 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:53.786195 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:53.799026 31636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:53.799050 31634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:53.799144 31633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:53.800390 31628 server_base.cc:1034] running on GCE node
I20260620 02:26:53.801520 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:53.803265 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:53.804570 31628 hybrid_clock.cc:648] HybridClock initialized: now 1781922413804516 us; error 51 us; skew 500 ppm
I20260620 02:26:53.805068 31628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:53 dist-test-slave-9vg1 krb5kdc[31615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:53.808857 31628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:53.810662 31628 webserver.cc:460] Webserver started at http://127.0.0.1:40249/ using document root <none> and password file <none>
I20260620 02:26:53.811514 31628 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:53.811682 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:53.812024 31628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:53.815593 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "bbca60e8066d48a6a9a0c6b990e499e9"
format_stamp: "Formatted at 2026-06-20 02:26:53 on dist-test-slave-9vg1"
server_key: "7a1a137bc4ac32192a4b212916e5af52"
server_key_iv: "f98808835b7a6eefec45acbd213f9717"
server_key_version: "encryptionkey@0"
I20260620 02:26:53.816500 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "bbca60e8066d48a6a9a0c6b990e499e9"
format_stamp: "Formatted at 2026-06-20 02:26:53 on dist-test-slave-9vg1"
server_key: "7a1a137bc4ac32192a4b212916e5af52"
server_key_iv: "f98808835b7a6eefec45acbd213f9717"
server_key_version: "encryptionkey@0"
I20260620 02:26:53.822672 31628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:26:53.826984 31643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:53.828847 31628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:26:53.829107 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "bbca60e8066d48a6a9a0c6b990e499e9"
format_stamp: "Formatted at 2026-06-20 02:26:53 on dist-test-slave-9vg1"
server_key: "7a1a137bc4ac32192a4b212916e5af52"
server_key_iv: "f98808835b7a6eefec45acbd213f9717"
server_key_version: "encryptionkey@0"
I20260620 02:26:53.829383 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:53.870376 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:53.876516 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:53.876837 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:53.898850 31628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38459
I20260620 02:26:53.898886 31694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38459 every 8 connection(s)
I20260620 02:26:53.900432 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:53.905040 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:53.907114 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31628
I20260620 02:26:53.907307 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:53.907871 10141 external_mini_cluster.cc:1428] Setting key 50303951ee86183300610b033ccf8578
I20260620 02:26:53.921371 31695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9: Bootstrap starting.
Jun 20 02:26:53 dist-test-slave-9vg1 krb5kdc[31615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:26:53.924949 31695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:53.926532 31695 log.cc:824] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:53.930413 31695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9: No bootstrap required, opened a new log
I20260620 02:26:53.934755 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:53.909886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:46595 (local address 127.0.0.1:38459)
0620 02:26:53.910814 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:53.910837 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:53.910895 (+ 58us) server_negotiation.cc:407] Connection header received
0620 02:26:53.912826 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:53.912866 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:53.913518 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:53.914021 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:53.915589 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:53.917715 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:53.919258 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:53.920435 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:53.920648 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:53.925177 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:53.925192 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:53.925216 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:53.925276 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:53.929382 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:53.930394 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:53.930406 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:53.930414 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:53.930664 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:53.931197 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:53.931203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:53.931207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:53.931972 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:53.932358 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:53.932738 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 02:26:53.933165 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":379,"threads_started":1}
I20260620 02:26:53.939625 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:53.941126 31695 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 02:26:53.941582 31695 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:53.941700 31695 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbca60e8066d48a6a9a0c6b990e499e9, State: Initialized, Role: FOLLOWER
I20260620 02:26:53.942488 31695 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 02:26:53.942807 31695 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:53.942953 31695 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:53.943186 31695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:53.946187 31695 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 02:26:53.946856 31695 leader_election.cc:302] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbca60e8066d48a6a9a0c6b990e499e9; no voters:
I20260620 02:26:53.947649 31695 leader_election.cc:288] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:53.947779 31705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:53.949069 31705 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [term 1 LEADER]: Becoming Leader. State: Replica: bbca60e8066d48a6a9a0c6b990e499e9, State: Running, Role: LEADER
I20260620 02:26:53.949738 31705 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 02:26:53.950302 31695 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:53.952922 31706 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } } }
I20260620 02:26:53.953437 31706 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:53.959972 31710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:53.961338 31707 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbca60e8066d48a6a9a0c6b990e499e9. Latest consensus state: current_term: 1 leader_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbca60e8066d48a6a9a0c6b990e499e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } } }
I20260620 02:26:53.961921 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:53.944530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:46657 (local address 127.0.0.1:38459)
0620 02:26:53.944719 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:53.944723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:53.945214 (+ 491us) server_negotiation.cc:407] Connection header received
0620 02:26:53.945417 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:53.945427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:53.945495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:53.945602 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:53.947338 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:53.948527 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:53.950150 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:53.950674 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:53.950808 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:53.954211 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:53.954217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:53.954220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:26:53.954256 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:53.957283 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:53.958165 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:53.958172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:53.958176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:53.958373 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:26:53.959151 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:26:53.959155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:26:53.959158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:26:53.959675 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:53.961033 (+ 1358us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:53.961295 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 02:26:53.961521 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:26:53.961978 31707 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:53.966444 31710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:53.977334 31710 catalog_manager.cc:1347] Generated new cluster ID: c697b716524d4b75949019fb5d68be50
I20260620 02:26:53.977478 31710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:54.017176 31710 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:54.019299 31710 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:54.034682 31710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbca60e8066d48a6a9a0c6b990e499e9: Generated new TSK 0
I20260620 02:26:54.036060 31710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:54.046829 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31628
I20260620 02:26:54.048337 31690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:26:54.146302 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31628
W20260620 02:26:54.164844 31613 connection.cc:535] client connection to 127.0.0.1:38459 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38459): Connection reset by peer (error 104)
2026-06-20T02:26:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2161 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:54 dist-test-slave-9vg1 krb5kdc[31727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:54 dist-test-slave-9vg1 krb5kdc[31727](info): set up 2 sockets
Jun 20 02:26:54 dist-test-slave-9vg1 krb5kdc[31727](info): commencing operation
krb5kdc: starting...
W20260620 02:26:55.316764 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:55 dist-test-slave-9vg1 krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:55.614178 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:55.979708 31740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:55.980026 31740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:55.980189 31740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:55.988209 31740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:55.988346 31740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:55.988440 31740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:55.988569 31740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:55.988631 31740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:55.999794 31740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:56.001259 31740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:56.002768 31740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:56.015805 31745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:56.015858 31746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:56.015834 31748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:56.016991 31740 server_base.cc:1034] running on GCE node
I20260620 02:26:56.017966 31740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:56.019677 31740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:56.020954 31740 hybrid_clock.cc:648] HybridClock initialized: now 1781922416020897 us; error 44 us; skew 500 ppm
I20260620 02:26:56.021384 31740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:56 dist-test-slave-9vg1 krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:56.025087 31740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:56.026975 31740 webserver.cc:460] Webserver started at http://127.0.0.1:37465/ using document root <none> and password file <none>
I20260620 02:26:56.027777 31740 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:56.027938 31740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:56.028262 31740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:56.031637 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "f512e72a1a1f42998cb30067247c432b"
format_stamp: "Formatted at 2026-06-20 02:26:56 on dist-test-slave-9vg1"
server_key: "7995a7bab8d7fcef9c062d54c48caae1"
server_key_iv: "403ec5d46c3bdfc67456bc35226e7f05"
server_key_version: "encryptionkey@0"
I20260620 02:26:56.032495 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "f512e72a1a1f42998cb30067247c432b"
format_stamp: "Formatted at 2026-06-20 02:26:56 on dist-test-slave-9vg1"
server_key: "7995a7bab8d7fcef9c062d54c48caae1"
server_key_iv: "403ec5d46c3bdfc67456bc35226e7f05"
server_key_version: "encryptionkey@0"
I20260620 02:26:56.038641 31740 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:26:56.042752 31755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:56.044441 31740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:26:56.044682 31740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "f512e72a1a1f42998cb30067247c432b"
format_stamp: "Formatted at 2026-06-20 02:26:56 on dist-test-slave-9vg1"
server_key: "7995a7bab8d7fcef9c062d54c48caae1"
server_key_iv: "403ec5d46c3bdfc67456bc35226e7f05"
server_key_version: "encryptionkey@0"
I20260620 02:26:56.044921 31740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:56.060968 31740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:56.066017 31740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:56.066347 31740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:56.086277 31740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43983
I20260620 02:26:56.086319 31806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43983 every 8 connection(s)
I20260620 02:26:56.087913 31740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:56.092139 31807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:56.094105 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31740
I20260620 02:26:56.094321 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:56.094811 10141 external_mini_cluster.cc:1428] Setting key 53bf8d9092fdd6c5b62c077eeea680cb
I20260620 02:26:56.106201 31807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b: Bootstrap starting.
I20260620 02:26:56.109938 31807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:56.110584 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:56.096956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:47637 (local address 127.0.0.1:43983)
0620 02:26:56.097856 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:56.097869 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:56.097921 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:26:56.100305 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:56.100333 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:56.100798 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:56.101161 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:56.101891 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:56.103479 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:56.105520 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:56.106338 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:56.106556 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:56.108078 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:56.108086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:56.108102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:56.108675 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:56.108762 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:56.108789 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:56.108962 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:56.109263 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 02:26:56.109451 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":391,"threads_started":1}
I20260620 02:26:56.111323 31807 log.cc:824] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:56.114737 31807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b: No bootstrap required, opened a new log
I20260620 02:26:56.115209 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:56.123610 31807 raft_consensus.cc:348] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f512e72a1a1f42998cb30067247c432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260620 02:26:56.124063 31807 raft_consensus.cc:374] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:56.124174 31807 raft_consensus.cc:729] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f512e72a1a1f42998cb30067247c432b, State: Initialized, Role: FOLLOWER
I20260620 02:26:56.124790 31807 consensus_queue.cc:260] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f512e72a1a1f42998cb30067247c432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260620 02:26:56.125077 31807 raft_consensus.cc:388] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:56.125240 31807 raft_consensus.cc:482] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:56.125449 31807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:56.127754 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:56.120698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:48295 (local address 127.0.0.1:43983)
0620 02:26:56.120886 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:56.120890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:56.121292 (+ 402us) server_negotiation.cc:407] Connection header received
0620 02:26:56.121548 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:56.121553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:56.121636 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:56.121763 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:56.122323 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:56.123206 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:56.124793 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:26:56.125361 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:26:56.125535 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:26:56.126899 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:56.126904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:56.126907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:56.126927 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:56.127002 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:56.127024 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:56.127210 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:56.127403 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 02:26:56.127461 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:26:56.128612 31807 raft_consensus.cc:504] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f512e72a1a1f42998cb30067247c432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260620 02:26:56.129271 31807 leader_election.cc:302] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f512e72a1a1f42998cb30067247c432b; no voters:
I20260620 02:26:56.129946 31807 leader_election.cc:288] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:56.130139 31817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:56.130970 31817 raft_consensus.cc:686] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [term 1 LEADER]: Becoming Leader. State: Replica: f512e72a1a1f42998cb30067247c432b, State: Running, Role: LEADER
I20260620 02:26:56.131618 31817 consensus_queue.cc:237] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f512e72a1a1f42998cb30067247c432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260620 02:26:56.132196 31807 sys_catalog.cc:564] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:56.134694 31819 sys_catalog.cc:455] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f512e72a1a1f42998cb30067247c432b. Latest consensus state: current_term: 1 leader_uuid: "f512e72a1a1f42998cb30067247c432b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f512e72a1a1f42998cb30067247c432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } } }
I20260620 02:26:56.134791 31818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f512e72a1a1f42998cb30067247c432b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f512e72a1a1f42998cb30067247c432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } } }
I20260620 02:26:56.135221 31819 sys_catalog.cc:458] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:56.135277 31818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:56.140468 31822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:56.146320 31822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:56.156916 31822 catalog_manager.cc:1347] Generated new cluster ID: 089a9f1381eb4c39bde9a8d1af04c2a9
I20260620 02:26:56.157063 31822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:56.166805 31835 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:26:56.182772 31822 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:56.184443 31822 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:56.192900 31822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f512e72a1a1f42998cb30067247c432b: Generated new TSK 0
I20260620 02:26:56.193866 31822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:56.209028 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31740
W20260620 02:26:56.210400 31802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.35:47637
W20260620 02:26:56.211592 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31740: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:56.211802 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31740
W20260620 02:26:56.228343 31725 connection.cc:535] client connection to 127.0.0.1:43983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43983): Connection reset by peer (error 104)
2026-06-20T02:26:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2063 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:56 dist-test-slave-9vg1 krb5kdc[31838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:56 dist-test-slave-9vg1 krb5kdc[31838](info): set up 2 sockets
Jun 20 02:26:56 dist-test-slave-9vg1 krb5kdc[31838](info): commencing operation
krb5kdc: starting...
W20260620 02:26:57.360737 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:57 dist-test-slave-9vg1 krb5kdc[31838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:26:57.651991 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:26:58.008044 31851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:26:58.008425 31851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:26:58.008610 31851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:26:58.016346 31851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:26:58.016526 31851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:26:58.016644 31851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:26:58.016757 31851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:26:58.016850 31851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:26:58.027698 31851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:26:58.029141 31851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:26:58.030551 31851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:26:58.043824 31857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:58.043889 31856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:26:58.043866 31859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:26:58.045411 31851 server_base.cc:1034] running on GCE node
I20260620 02:26:58.046443 31851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:26:58.048194 31851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:26:58.049453 31851 hybrid_clock.cc:648] HybridClock initialized: now 1781922418049389 us; error 54 us; skew 500 ppm
I20260620 02:26:58.049866 31851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:26:58 dist-test-slave-9vg1 krb5kdc[31838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:26:58.053551 31851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:26:58.055223 31851 webserver.cc:460] Webserver started at http://127.0.0.1:36921/ using document root <none> and password file <none>
I20260620 02:26:58.055984 31851 fs_manager.cc:359] Metadata directory not provided
I20260620 02:26:58.056102 31851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:26:58.056394 31851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:26:58.059854 31851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "639f11ca02074e068406cfe65ffd7cf6"
format_stamp: "Formatted at 2026-06-20 02:26:58 on dist-test-slave-9vg1"
server_key: "f2c64256902688af149475b4932c4ed4"
server_key_iv: "1185c65999509c7780c3425dff9d2060"
server_key_version: "encryptionkey@0"
I20260620 02:26:58.060685 31851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "639f11ca02074e068406cfe65ffd7cf6"
format_stamp: "Formatted at 2026-06-20 02:26:58 on dist-test-slave-9vg1"
server_key: "f2c64256902688af149475b4932c4ed4"
server_key_iv: "1185c65999509c7780c3425dff9d2060"
server_key_version: "encryptionkey@0"
I20260620 02:26:58.066761 31851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:26:58.070789 31866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:26:58.072436 31851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:26:58.072692 31851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "639f11ca02074e068406cfe65ffd7cf6"
format_stamp: "Formatted at 2026-06-20 02:26:58 on dist-test-slave-9vg1"
server_key: "f2c64256902688af149475b4932c4ed4"
server_key_iv: "1185c65999509c7780c3425dff9d2060"
server_key_version: "encryptionkey@0"
I20260620 02:26:58.072923 31851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:26:58.079144 31851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:26:58.083717 31851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:26:58.084074 31851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:26:58.104650 31851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41371
I20260620 02:26:58.104754 31917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41371 every 8 connection(s)
I20260620 02:26:58.106218 31851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:26:58.110103 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31851
I20260620 02:26:58.110313 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:26:58.110459 31918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:26:58.110865 10141 external_mini_cluster.cc:1428] Setting key d8ec687cba0ca2853ebe5f9eb90664fe
I20260620 02:26:58.119848 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:58.112985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:33075 (local address 127.0.0.1:41371)
0620 02:26:58.113974 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:58.113986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:58.114034 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:26:58.115783 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:58.115807 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:58.116183 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:58.116549 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:58.118007 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:58.118014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:58.118031 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:58.118053 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:58.118139 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:58.118162 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:58.118362 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:58.118474 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 02:26:58.118775 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":431,"threads_started":1}
I20260620 02:26:58.123912 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:26:58.124907 31918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6: Bootstrap starting.
I20260620 02:26:58.128439 31918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6: Neither blocks nor log segments found. Creating new log.
I20260620 02:26:58.130002 31918 log.cc:824] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:26:58.131562 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:26:58.128349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:36607 (local address 127.0.0.1:41371)
0620 02:26:58.128593 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:26:58.128602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:26:58.129196 (+ 594us) server_negotiation.cc:407] Connection header received
0620 02:26:58.129407 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:26:58.129412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:26:58.129468 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:26:58.129561 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:26:58.130751 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:26:58.130759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:26:58.130769 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:26:58.130790 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:26:58.130853 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:26:58.130875 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:26:58.131015 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:26:58.131151 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 02:26:58.131224 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:26:58.133581 31918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6: No bootstrap required, opened a new log
I20260620 02:26:58.142059 31918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639f11ca02074e068406cfe65ffd7cf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 02:26:58.142541 31918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:26:58.142668 31918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639f11ca02074e068406cfe65ffd7cf6, State: Initialized, Role: FOLLOWER
I20260620 02:26:58.143397 31918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639f11ca02074e068406cfe65ffd7cf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 02:26:58.143667 31918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:26:58.143867 31918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:26:58.144117 31918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:26:58.146746 31918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639f11ca02074e068406cfe65ffd7cf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 02:26:58.147360 31918 leader_election.cc:302] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 639f11ca02074e068406cfe65ffd7cf6; no voters:
I20260620 02:26:58.148003 31918 leader_election.cc:288] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:26:58.148173 31928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:26:58.148873 31928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [term 1 LEADER]: Becoming Leader. State: Replica: 639f11ca02074e068406cfe65ffd7cf6, State: Running, Role: LEADER
I20260620 02:26:58.149549 31928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639f11ca02074e068406cfe65ffd7cf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 02:26:58.150359 31918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:26:58.152807 31930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 639f11ca02074e068406cfe65ffd7cf6. Latest consensus state: current_term: 1 leader_uuid: "639f11ca02074e068406cfe65ffd7cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639f11ca02074e068406cfe65ffd7cf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } } }
I20260620 02:26:58.152781 31929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "639f11ca02074e068406cfe65ffd7cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639f11ca02074e068406cfe65ffd7cf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } } }
I20260620 02:26:58.153311 31930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:58.153337 31929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:26:58.154449 31935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:26:58.161412 31935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:26:58.178282 31935 catalog_manager.cc:1347] Generated new cluster ID: 5797ff803be748fbbc8b322246f31760
I20260620 02:26:58.178429 31935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:26:58.195374 31935 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:26:58.195863 31935 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:26:58.203140 31935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 639f11ca02074e068406cfe65ffd7cf6: Generated new TSK 0
I20260620 02:26:58.204142 31935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:26:58.219928 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31851
W20260620 02:26:58.221467 31913 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.35:33075
W20260620 02:26:58.222344 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:26:58.222517 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31851
2026-06-20T02:26:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:26:58 dist-test-slave-9vg1 krb5kdc[31949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:26:58 dist-test-slave-9vg1 krb5kdc[31949](info): set up 2 sockets
Jun 20 02:26:58 dist-test-slave-9vg1 krb5kdc[31949](info): commencing operation
krb5kdc: starting...
W20260620 02:26:59.364677 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:26:59 dist-test-slave-9vg1 krb5kdc[31949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:26:59.650803 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:32943
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:00.007920 31962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:00.008245 31962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:00.008416 31962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:00.016325 31962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:00.016474 31962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:00.016577 31962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:00.016708 31962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:00.016775 31962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:00.028077 31962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:32943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:32943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:00.029580 31962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:00.030992 31962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:00.044147 31968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:00.044165 31970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:00.044184 31967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:00.045157 31962 server_base.cc:1034] running on GCE node
I20260620 02:27:00.045964 31962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:00.047698 31962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:00.048947 31962 hybrid_clock.cc:648] HybridClock initialized: now 1781922420048902 us; error 54 us; skew 500 ppm
I20260620 02:27:00.049355 31962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:00 dist-test-slave-9vg1 krb5kdc[31949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922420, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:00.057545 31962 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:00.059468 31962 webserver.cc:460] Webserver started at http://127.9.231.126:34771/ using document root <none> and password file <none>
I20260620 02:27:00.060240 31962 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:00.060392 31962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:00.060724 31962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:00.064054 31962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "071ce076560c41539bdedb0210071328"
format_stamp: "Formatted at 2026-06-20 02:27:00 on dist-test-slave-9vg1"
server_key: "b28b83e82618fcb33152456cac502532"
server_key_iv: "ab588b5ebe35a3696edd05f3f2fdd202"
server_key_version: "encryptionkey@0"
I20260620 02:27:00.064924 31962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "071ce076560c41539bdedb0210071328"
format_stamp: "Formatted at 2026-06-20 02:27:00 on dist-test-slave-9vg1"
server_key: "b28b83e82618fcb33152456cac502532"
server_key_iv: "ab588b5ebe35a3696edd05f3f2fdd202"
server_key_version: "encryptionkey@0"
I20260620 02:27:00.070542 31962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:27:00.074406 31977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:00.075973 31962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:27:00.076216 31962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "071ce076560c41539bdedb0210071328"
format_stamp: "Formatted at 2026-06-20 02:27:00 on dist-test-slave-9vg1"
server_key: "b28b83e82618fcb33152456cac502532"
server_key_iv: "ab588b5ebe35a3696edd05f3f2fdd202"
server_key_version: "encryptionkey@0"
I20260620 02:27:00.076439 31962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:00.093191 31962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:00.098268 31962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:00.098618 31962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:00.119369 31962 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:32943
I20260620 02:27:00.119323 32028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:32943 every 8 connection(s)
I20260620 02:27:00.121100 31962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:00.125540 32029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:00.129140 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 31962
I20260620 02:27:00.129361 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:00.129916 10141 external_mini_cluster.cc:1428] Setting key 98a1a9c20c32d6991b786f46867a0f18
I20260620 02:27:00.138751 32029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328: Bootstrap starting.
I20260620 02:27:00.142129 32029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:00.143309 32029 log.cc:824] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:27:00 dist-test-slave-9vg1 krb5kdc[31949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:27:00.146920 32029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328: No bootstrap required, opened a new log
I20260620 02:27:00.155525 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:00.132026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53217 (local address 127.9.231.126:32943)
0620 02:27:00.132962 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:00.132974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:00.133020 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:27:00.134867 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:00.134893 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:00.135450 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:00.135936 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:00.137604 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:00.139218 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:00.140637 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:00.141401 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:00.141639 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:00.146935 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:00.146944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:00.146959 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:00.147050 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:00.150450 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:00.151550 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:00.151557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:00.151564 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:00.151798 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:00.152380 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:00.152384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:00.152386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:00.153000 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:00.153282 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:00.153847 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 02:27:00.154260 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":448,"threads_started":1}
I20260620 02:27:00.156626 32029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071ce076560c41539bdedb0210071328" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32943 } }
I20260620 02:27:00.157018 32029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:27:00.157132 32029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071ce076560c41539bdedb0210071328, State: Initialized, Role: FOLLOWER
I20260620 02:27:00.157866 32029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071ce076560c41539bdedb0210071328" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32943 } }
I20260620 02:27:00.158149 32029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:27:00.158313 32029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:27:00.158492 32029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:00.159857 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:00.161700 32029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071ce076560c41539bdedb0210071328" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32943 } }
I20260620 02:27:00.162313 32029 leader_election.cc:302] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 071ce076560c41539bdedb0210071328; no voters:
I20260620 02:27:00.163081 32029 leader_election.cc:288] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:27:00.163285 32034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:00.164021 32034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [term 1 LEADER]: Becoming Leader. State: Replica: 071ce076560c41539bdedb0210071328, State: Running, Role: LEADER
I20260620 02:27:00.164664 32034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071ce076560c41539bdedb0210071328" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32943 } }
I20260620 02:27:00.166674 32034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 071ce076560c41539bdedb0210071328. Latest consensus state: current_term: 1 leader_uuid: "071ce076560c41539bdedb0210071328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071ce076560c41539bdedb0210071328" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32943 } } }
I20260620 02:27:00.166996 32034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:00.167659 32038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "071ce076560c41539bdedb0210071328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071ce076560c41539bdedb0210071328" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 32943 } } }
I20260620 02:27:00.168114 32038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:00.176731 32029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:00.177171 32040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:00.187927 32040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:27:00.201538 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:00.176834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57105 (local address 127.9.231.126:32943)
0620 02:27:00.177038 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:00.177043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:00.179765 (+ 2722us) server_negotiation.cc:407] Connection header received
0620 02:27:00.180040 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:00.180046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:00.180129 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:00.180267 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:00.182442 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:00.183513 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:00.185529 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:00.186352 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:00.186485 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:00.190606 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:00.190616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:00.190622 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:00.190673 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:00.195738 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:00.197262 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:00.197276 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:00.197281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:00.197486 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:00.198241 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:00.198249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:00.198255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:00.198831 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:00.199198 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:00.200699 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0620 02:27:00.200961 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 02:27:00.206738 32040 catalog_manager.cc:1347] Generated new cluster ID: 1d71fde7c64e4472b3ed5abe57fadb14
I20260620 02:27:00.206884 32040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:00.242617 32040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:00.244263 32040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:00.253350 32040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 071ce076560c41539bdedb0210071328: Generated new TSK 0
I20260620 02:27:00.254384 32040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:27:00.271631 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 31962
I20260620 02:27:00.273064 32024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:00.371343 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 31962
W20260620 02:27:00.389946 31947 connection.cc:535] client connection to 127.9.231.126:32943 recv error: Network error: failed to read from TLS socket (remote: 127.9.231.126:32943): Connection reset by peer (error 104)
2026-06-20T02:27:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2149 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:27:00 dist-test-slave-9vg1 krb5kdc[32061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:27:00 dist-test-slave-9vg1 krb5kdc[32061](info): set up 2 sockets
Jun 20 02:27:00 dist-test-slave-9vg1 krb5kdc[32061](info): commencing operation
krb5kdc: starting...
W20260620 02:27:01.516731 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:27:01 dist-test-slave-9vg1 krb5kdc[32061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:27:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:27:01.795881 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38483
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:02.147680 32074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:02.148072 32074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:02.148247 32074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:02.155915 32074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:02.156100 32074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:02.156219 32074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:02.156329 32074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:02.156407 32074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:02.166673 32074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:02.168156 32074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:02.169554 32074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:02.183430 32079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:02.183526 32082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:02.183389 32080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:02.184582 32074 server_base.cc:1034] running on GCE node
I20260620 02:27:02.185461 32074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:02.187044 32074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:02.188366 32074 hybrid_clock.cc:648] HybridClock initialized: now 1781922422188284 us; error 85 us; skew 500 ppm
I20260620 02:27:02.188831 32074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:02 dist-test-slave-9vg1 krb5kdc[32061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922422, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:02.192384 32074 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:02.194377 32074 webserver.cc:460] Webserver started at http://127.9.231.126:34345/ using document root <none> and password file <none>
I20260620 02:27:02.195235 32074 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:02.195389 32074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:02.195773 32074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:02.199676 32074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "b1f314051f71458b9d294de0a9f3c6f5"
format_stamp: "Formatted at 2026-06-20 02:27:02 on dist-test-slave-9vg1"
server_key: "0f8f596e955615f600b30c3ad6c52753"
server_key_iv: "e13be61bd589c86f05f4110122d41fa2"
server_key_version: "encryptionkey@0"
I20260620 02:27:02.200636 32074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "b1f314051f71458b9d294de0a9f3c6f5"
format_stamp: "Formatted at 2026-06-20 02:27:02 on dist-test-slave-9vg1"
server_key: "0f8f596e955615f600b30c3ad6c52753"
server_key_iv: "e13be61bd589c86f05f4110122d41fa2"
server_key_version: "encryptionkey@0"
I20260620 02:27:02.206828 32074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260620 02:27:02.210989 32089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:02.212711 32074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:27:02.212951 32074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "b1f314051f71458b9d294de0a9f3c6f5"
format_stamp: "Formatted at 2026-06-20 02:27:02 on dist-test-slave-9vg1"
server_key: "0f8f596e955615f600b30c3ad6c52753"
server_key_iv: "e13be61bd589c86f05f4110122d41fa2"
server_key_version: "encryptionkey@0"
I20260620 02:27:02.213194 32074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:02.226071 32074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:02.231432 32074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:02.231786 32074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:02.254477 32074 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38483
I20260620 02:27:02.254421 32140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38483 every 8 connection(s)
I20260620 02:27:02.256229 32074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:02.260466 32141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:02.264958 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 32074
I20260620 02:27:02.265210 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:02.265703 10141 external_mini_cluster.cc:1428] Setting key 25a57344bf7c3fdc2a992610fcef0d79
I20260620 02:27:02.272821 32141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5: Bootstrap starting.
I20260620 02:27:02.276861 32141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:02.278113 32141 log.cc:824] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:02.281471 32144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:02.267908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41627 (local address 127.9.231.126:38483)
0620 02:27:02.268908 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:02.268921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:02.268968 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:27:02.271264 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:02.271296 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:02.271856 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:02.272309 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:02.272946 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:02.274636 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:02.276336 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:02.277195 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:02.277430 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:02.278858 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:02.278867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:02.278884 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:02.279618 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:02.279721 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:02.279749 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:02.279927 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:02.280243 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:27:02.280438 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":508,"threads_started":1}
I20260620 02:27:02.281677 32141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5: No bootstrap required, opened a new log
I20260620 02:27:02.286208 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:02.291096 32141 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f314051f71458b9d294de0a9f3c6f5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38483 } }
I20260620 02:27:02.291644 32141 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:27:02.291760 32141 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f314051f71458b9d294de0a9f3c6f5, State: Initialized, Role: FOLLOWER
I20260620 02:27:02.292616 32141 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f314051f71458b9d294de0a9f3c6f5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38483 } }
I20260620 02:27:02.292964 32141 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:27:02.293135 32141 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:27:02.293347 32141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:02.296483 32141 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f314051f71458b9d294de0a9f3c6f5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38483 } }
I20260620 02:27:02.297237 32141 leader_election.cc:302] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f314051f71458b9d294de0a9f3c6f5; no voters:
I20260620 02:27:02.298005 32141 leader_election.cc:288] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:27:02.298159 32151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:02.300315 32144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:02.291465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45011 (local address 127.9.231.126:38483)
0620 02:27:02.291803 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:02.291811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:02.292289 (+ 478us) server_negotiation.cc:407] Connection header received
0620 02:27:02.292490 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:02.292495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:02.292583 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:02.292717 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:02.293442 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:02.294745 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:02.296402 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:02.297095 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:02.297246 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:02.298796 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:02.298808 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:02.298817 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:02.298863 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:02.298944 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:02.298977 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:02.299150 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:02.299588 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 02:27:02.299641 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":135}
I20260620 02:27:02.301035 32141 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:02.302783 32151 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [term 1 LEADER]: Becoming Leader. State: Replica: b1f314051f71458b9d294de0a9f3c6f5, State: Running, Role: LEADER
I20260620 02:27:02.303476 32151 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f314051f71458b9d294de0a9f3c6f5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38483 } }
I20260620 02:27:02.307037 32152 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f314051f71458b9d294de0a9f3c6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f314051f71458b9d294de0a9f3c6f5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38483 } } }
I20260620 02:27:02.307580 32152 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:02.308135 32153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f314051f71458b9d294de0a9f3c6f5. Latest consensus state: current_term: 1 leader_uuid: "b1f314051f71458b9d294de0a9f3c6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f314051f71458b9d294de0a9f3c6f5" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38483 } } }
I20260620 02:27:02.308462 32153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:02.308701 32160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:02.315807 32160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:27:02.328670 32160 catalog_manager.cc:1347] Generated new cluster ID: 05605a1bc6bf447281f278891558a736
I20260620 02:27:02.328898 32160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:02.345976 32160 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:02.347549 32160 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:02.364675 32160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f314051f71458b9d294de0a9f3c6f5: Generated new TSK 0
I20260620 02:27:02.365921 32160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:27:02.380872 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32074
W20260620 02:27:02.382495 32136 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41627
W20260620 02:27:02.383548 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:27:02.383733 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 32074
W20260620 02:27:02.404068 32059 connection.cc:535] client connection to 127.9.231.126:38483 recv error: Network error: failed to read from TLS socket (remote: 127.9.231.126:38483): Connection reset by peer (error 104)
2026-06-20T02:27:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:27:02 dist-test-slave-9vg1 krb5kdc[32172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:27:02 dist-test-slave-9vg1 krb5kdc[32172](info): set up 2 sockets
Jun 20 02:27:02 dist-test-slave-9vg1 krb5kdc[32172](info): commencing operation
krb5kdc: starting...
W20260620 02:27:03.557292 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:27:03 dist-test-slave-9vg1 krb5kdc[32172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:27:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:27:03.844763 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34441
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:34441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:04.207548 32185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:04.207901 32185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:04.208069 32185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:04.215821 32185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:04.215953 32185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:04.216044 32185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:04.216145 32185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:04.216212 32185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:04.226826 32185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:34441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:04.228407 32185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:04.229799 32185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:04.243242 32191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:04.243330 32193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:04.243367 32190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:04.244808 32185 server_base.cc:1034] running on GCE node
I20260620 02:27:04.245682 32185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:04.247341 32185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:04.248574 32185 hybrid_clock.cc:648] HybridClock initialized: now 1781922424248500 us; error 68 us; skew 500 ppm
I20260620 02:27:04.248935 32185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:04 dist-test-slave-9vg1 krb5kdc[32172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922424, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:04.252458 32185 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:04.254078 32185 webserver.cc:460] Webserver started at http://127.9.231.126:46831/ using document root <none> and password file <none>
I20260620 02:27:04.254793 32185 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:04.254900 32185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:04.255251 32185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:04.258489 32185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "08188de873944fbf9828705588ccbf2e"
format_stamp: "Formatted at 2026-06-20 02:27:04 on dist-test-slave-9vg1"
server_key: "57fc11974dc7bc3b6b44ba4543907e70"
server_key_iv: "801620633e6006d05970641002ce9c18"
server_key_version: "encryptionkey@0"
I20260620 02:27:04.259341 32185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "08188de873944fbf9828705588ccbf2e"
format_stamp: "Formatted at 2026-06-20 02:27:04 on dist-test-slave-9vg1"
server_key: "57fc11974dc7bc3b6b44ba4543907e70"
server_key_iv: "801620633e6006d05970641002ce9c18"
server_key_version: "encryptionkey@0"
I20260620 02:27:04.264914 32185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:27:04.268769 32200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:04.270402 32185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:27:04.270648 32185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "08188de873944fbf9828705588ccbf2e"
format_stamp: "Formatted at 2026-06-20 02:27:04 on dist-test-slave-9vg1"
server_key: "57fc11974dc7bc3b6b44ba4543907e70"
server_key_iv: "801620633e6006d05970641002ce9c18"
server_key_version: "encryptionkey@0"
I20260620 02:27:04.270903 32185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:04.277608 32185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:04.282584 32185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:04.282948 32185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:04.302785 32185 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:34441
I20260620 02:27:04.302820 32251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:34441 every 8 connection(s)
I20260620 02:27:04.304368 32185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:04.308409 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:04.314132 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 32185
I20260620 02:27:04.314332 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:04.314880 10141 external_mini_cluster.cc:1428] Setting key 7dd63bbd67ed9611416e906f69ba545a
I20260620 02:27:04.321138 32252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e: Bootstrap starting.
I20260620 02:27:04.325403 32255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:04.317270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37737 (local address 127.9.231.126:34441)
0620 02:27:04.318371 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:04.318392 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:04.318803 (+ 411us) server_negotiation.cc:407] Connection header received
0620 02:27:04.321287 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:04.321320 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:04.321817 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:04.322267 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:04.323384 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:04.323393 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:04.323411 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:04.323435 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:04.323512 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:04.323555 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:04.323798 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:04.323924 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 02:27:04.324128 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":516,"threads_started":1}
I20260620 02:27:04.325549 32252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:04.326769 32252 log.cc:824] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:04.329818 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:04.330289 32252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e: No bootstrap required, opened a new log
I20260620 02:27:04.338459 32255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:04.335373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54485 (local address 127.9.231.126:34441)
0620 02:27:04.335581 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:04.335588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:04.336191 (+ 603us) server_negotiation.cc:407] Connection header received
0620 02:27:04.336340 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:04.336347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:04.336423 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:04.336553 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:04.337654 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:04.337661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:04.337666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:04.337687 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:04.337747 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:04.337773 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:04.337908 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:04.338052 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 02:27:04.338116 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:27:04.340612 32252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08188de873944fbf9828705588ccbf2e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34441 } }
I20260620 02:27:04.341012 32252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:27:04.341152 32252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08188de873944fbf9828705588ccbf2e, State: Initialized, Role: FOLLOWER
I20260620 02:27:04.341841 32252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08188de873944fbf9828705588ccbf2e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34441 } }
I20260620 02:27:04.342108 32252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:27:04.342265 32252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:27:04.342458 32252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:04.344908 32252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08188de873944fbf9828705588ccbf2e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34441 } }
I20260620 02:27:04.345531 32252 leader_election.cc:302] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08188de873944fbf9828705588ccbf2e; no voters:
I20260620 02:27:04.346179 32252 leader_election.cc:288] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:27:04.346326 32262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:04.347116 32262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [term 1 LEADER]: Becoming Leader. State: Replica: 08188de873944fbf9828705588ccbf2e, State: Running, Role: LEADER
I20260620 02:27:04.347681 32262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08188de873944fbf9828705588ccbf2e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34441 } }
I20260620 02:27:04.348637 32252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:04.350930 32264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08188de873944fbf9828705588ccbf2e. Latest consensus state: current_term: 1 leader_uuid: "08188de873944fbf9828705588ccbf2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08188de873944fbf9828705588ccbf2e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34441 } } }
I20260620 02:27:04.351433 32264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:04.351320 32263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08188de873944fbf9828705588ccbf2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08188de873944fbf9828705588ccbf2e" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 34441 } } }
I20260620 02:27:04.351619 32263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:04.352300 32268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:04.359431 32268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:27:04.369750 32268 catalog_manager.cc:1347] Generated new cluster ID: 6f81654f5c9447e280c5c9c507c13679
I20260620 02:27:04.369901 32268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:04.409832 32268 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:04.410363 32268 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:04.426090 32268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08188de873944fbf9828705588ccbf2e: Generated new TSK 0
I20260620 02:27:04.427342 32268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:27:04.432659 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32185
W20260620 02:27:04.434168 32247 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37737
W20260620 02:27:04.435087 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:27:04.435261 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 32185
2026-06-20T02:27:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2046 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:27:04 dist-test-slave-9vg1 krb5kdc[32283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:27:04 dist-test-slave-9vg1 krb5kdc[32283](info): set up 2 sockets
Jun 20 02:27:04 dist-test-slave-9vg1 krb5kdc[32283](info): commencing operation
krb5kdc: starting...
W20260620 02:27:05.604782 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:27:05 dist-test-slave-9vg1 krb5kdc[32283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:27:05.891238 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:43591
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:06.265975 32296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:06.266398 32296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:06.266618 32296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:06.274513 32296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:06.274663 32296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:06.274789 32296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:06.274887 32296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:06.274981 32296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:06.286051 32296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:06.287576 32296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:06.288988 32296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:06.302371 32301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:06.302309 32302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:06.302395 32304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:06.303534 32296 server_base.cc:1034] running on GCE node
I20260620 02:27:06.304412 32296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:06.305997 32296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:06.307276 32296 hybrid_clock.cc:648] HybridClock initialized: now 1781922426307230 us; error 54 us; skew 500 ppm
I20260620 02:27:06.307768 32296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:06 dist-test-slave-9vg1 krb5kdc[32283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922426, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:06.316529 32296 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:06.318359 32296 webserver.cc:460] Webserver started at http://127.9.231.126:46793/ using document root <none> and password file <none>
I20260620 02:27:06.319235 32296 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:06.319396 32296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:06.319728 32296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:06.323119 32296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "3c1cacec0b814e52841950526b3095b1"
format_stamp: "Formatted at 2026-06-20 02:27:06 on dist-test-slave-9vg1"
server_key: "36abd184f47d3f6ea678e6e95029978b"
server_key_iv: "bd5a2e0e63e1f8b369e54dbb993ecdec"
server_key_version: "encryptionkey@0"
I20260620 02:27:06.324007 32296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "3c1cacec0b814e52841950526b3095b1"
format_stamp: "Formatted at 2026-06-20 02:27:06 on dist-test-slave-9vg1"
server_key: "36abd184f47d3f6ea678e6e95029978b"
server_key_iv: "bd5a2e0e63e1f8b369e54dbb993ecdec"
server_key_version: "encryptionkey@0"
I20260620 02:27:06.329923 32296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:27:06.334240 32311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:06.335841 32296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:27:06.336059 32296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "3c1cacec0b814e52841950526b3095b1"
format_stamp: "Formatted at 2026-06-20 02:27:06 on dist-test-slave-9vg1"
server_key: "36abd184f47d3f6ea678e6e95029978b"
server_key_iv: "bd5a2e0e63e1f8b369e54dbb993ecdec"
server_key_version: "encryptionkey@0"
I20260620 02:27:06.336292 32296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:06.383220 32296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:06.388228 32296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:06.388559 32296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:06.408737 32296 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:43591
I20260620 02:27:06.408757 32362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:43591 every 8 connection(s)
I20260620 02:27:06.410220 32296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:06.411664 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 32296
I20260620 02:27:06.411861 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:06.412463 10141 external_mini_cluster.cc:1428] Setting key 1c81fbaede5715448c52ccc37a03bda1
I20260620 02:27:06.415511 32363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:27:06 dist-test-slave-9vg1 krb5kdc[32283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:27:06.427268 32363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1: Bootstrap starting.
I20260620 02:27:06.431555 32363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:06.432960 32363 log.cc:824] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:06.436723 32363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1: No bootstrap required, opened a new log
I20260620 02:27:06.437160 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:06.414499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60373 (local address 127.9.231.126:43591)
0620 02:27:06.415359 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:06.415373 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:06.415414 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:27:06.417165 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:06.417189 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:06.417594 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:06.418018 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:06.419677 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:06.421075 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:06.422667 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:06.423613 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:06.423869 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:06.428478 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:06.428487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:06.428505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:06.428557 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:06.432188 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:06.433214 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:06.433222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:06.433229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:06.433418 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:06.433930 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:06.433937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:06.433940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:06.434700 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:06.434991 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:06.435510 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 02:27:06.435949 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":362,"threads_started":1}
I20260620 02:27:06.441555 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:06.447649 32363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1cacec0b814e52841950526b3095b1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43591 } }
I20260620 02:27:06.448088 32363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:27:06.448227 32363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1cacec0b814e52841950526b3095b1, State: Initialized, Role: FOLLOWER
I20260620 02:27:06.448935 32363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1cacec0b814e52841950526b3095b1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43591 } }
I20260620 02:27:06.449200 32363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:27:06.449348 32363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:27:06.449553 32363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:06.452036 32363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1cacec0b814e52841950526b3095b1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43591 } }
I20260620 02:27:06.452631 32363 leader_election.cc:302] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c1cacec0b814e52841950526b3095b1; no voters:
I20260620 02:27:06.453372 32363 leader_election.cc:288] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:27:06.453528 32373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:06.454671 32373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [term 1 LEADER]: Becoming Leader. State: Replica: 3c1cacec0b814e52841950526b3095b1, State: Running, Role: LEADER
I20260620 02:27:06.455335 32373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1cacec0b814e52841950526b3095b1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43591 } }
I20260620 02:27:06.455897 32363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:06.458849 32374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c1cacec0b814e52841950526b3095b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1cacec0b814e52841950526b3095b1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43591 } } }
I20260620 02:27:06.458887 32375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c1cacec0b814e52841950526b3095b1. Latest consensus state: current_term: 1 leader_uuid: "3c1cacec0b814e52841950526b3095b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1cacec0b814e52841950526b3095b1" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 43591 } } }
I20260620 02:27:06.459362 32374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:06.459362 32375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:06.461459 32380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:06.466528 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:06.447319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51333 (local address 127.9.231.126:43591)
0620 02:27:06.447531 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:06.447537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:06.448180 (+ 643us) server_negotiation.cc:407] Connection header received
0620 02:27:06.448389 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:06.448395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:06.448471 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:06.448673 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:06.450423 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:06.451287 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:06.452610 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:06.453136 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:06.453286 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:06.456351 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:06.456357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:06.456360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:06.456395 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:06.461087 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:06.462140 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:06.462147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:06.462150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:06.462425 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:06.463264 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:06.463269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:06.463272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:06.463705 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:06.464099 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:06.465620 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0620 02:27:06.465946 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:27:06.473801 32380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:27:06.484033 32380 catalog_manager.cc:1347] Generated new cluster ID: cabc84a0a7344ccbb516699cb9307fdd
I20260620 02:27:06.484187 32380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:06.497485 32380 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:06.498926 32380 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:06.507040 32380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c1cacec0b814e52841950526b3095b1: Generated new TSK 0
I20260620 02:27:06.508020 32380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:27:06.518321 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32296
I20260620 02:27:06.519620 32358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:06.614993 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 32296
2026-06-20T02:27:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:27:06 dist-test-slave-9vg1 krb5kdc[32395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:27:06 dist-test-slave-9vg1 krb5kdc[32395](info): set up 2 sockets
Jun 20 02:27:06 dist-test-slave-9vg1 krb5kdc[32395](info): commencing operation
krb5kdc: starting...
W20260620 02:27:07.744884 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:27:07 dist-test-slave-9vg1 krb5kdc[32395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:27:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:27:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:27:08.030767 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33549
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:37667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:08.385262 32408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:08.385663 32408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:08.385875 32408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:08.394447 32408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:08.394594 32408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:08.394691 32408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:08.394830 32408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:08.394903 32408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:08.406394 32408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:37667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:33549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:08.408111 32408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:08.409657 32408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:08.423342 32414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:08.423769 32413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:08.423965 32416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:08.424360 32408 server_base.cc:1034] running on GCE node
I20260620 02:27:08.425235 32408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:08.426771 32408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:08.428027 32408 hybrid_clock.cc:648] HybridClock initialized: now 1781922428427947 us; error 73 us; skew 500 ppm
I20260620 02:27:08.428501 32408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:08 dist-test-slave-9vg1 krb5kdc[32395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922428, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:08.431998 32408 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:08.433627 32408 webserver.cc:460] Webserver started at http://127.9.231.126:35531/ using document root <none> and password file <none>
I20260620 02:27:08.434413 32408 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:08.434559 32408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:08.434896 32408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:08.438249 32408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8"
format_stamp: "Formatted at 2026-06-20 02:27:08 on dist-test-slave-9vg1"
server_key: "379430fa66d05d91f63bb73b6d2984c3"
server_key_iv: "19923f3a87cc573bca63322454f32a96"
server_key_version: "encryptionkey@0"
I20260620 02:27:08.439168 32408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8"
format_stamp: "Formatted at 2026-06-20 02:27:08 on dist-test-slave-9vg1"
server_key: "379430fa66d05d91f63bb73b6d2984c3"
server_key_iv: "19923f3a87cc573bca63322454f32a96"
server_key_version: "encryptionkey@0"
I20260620 02:27:08.444661 32408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:27:08.448665 32423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:08.450135 32408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:27:08.450325 32408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8"
format_stamp: "Formatted at 2026-06-20 02:27:08 on dist-test-slave-9vg1"
server_key: "379430fa66d05d91f63bb73b6d2984c3"
server_key_iv: "19923f3a87cc573bca63322454f32a96"
server_key_version: "encryptionkey@0"
I20260620 02:27:08.450534 32408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:08.470337 32408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:08.474965 32408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:08.475328 32408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:08.495671 32408 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:33549
I20260620 02:27:08.495703 32474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:33549 every 8 connection(s)
I20260620 02:27:08.497193 32408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:08.499457 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 32408
I20260620 02:27:08.499761 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:08.500217 10141 external_mini_cluster.cc:1428] Setting key 1dbe1ad04cfa77bbdc119d114703aee9
I20260620 02:27:08.501809 32475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:08.516017 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:08.502080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54595 (local address 127.9.231.126:33549)
0620 02:27:08.503364 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:08.503382 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:08.503441 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:27:08.505843 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:08.505869 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:08.506340 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:08.506720 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:08.507428 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:08.508850 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:08.510828 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:08.511719 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:08.511861 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:08.513250 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:08.513258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:08.513277 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:08.513881 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:08.513975 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:08.514006 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:08.514116 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:08.514453 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 02:27:08.514888 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":725,"threads_started":1}
I20260620 02:27:08.517428 32475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8: Bootstrap starting.
I20260620 02:27:08.520244 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:08.521075 32475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:08.522205 32475 log.cc:824] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:08.526270 32475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8: No bootstrap required, opened a new log
I20260620 02:27:08.533577 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:08.525704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58435 (local address 127.9.231.126:33549)
0620 02:27:08.525953 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:08.525960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:08.526448 (+ 488us) server_negotiation.cc:407] Connection header received
0620 02:27:08.526638 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:08.526647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:08.526737 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:08.526885 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:08.527427 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:08.528654 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:08.530024 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:08.530858 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:08.531015 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:08.532600 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:08.532607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:08.532613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:08.532648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:08.532709 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:08.532732 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:08.532849 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:08.533033 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 02:27:08.533156 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:27:08.537571 32475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33549 } }
I20260620 02:27:08.538031 32475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:27:08.538115 32475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3dfd8ea7944196ab5f49b23d4ca8b8, State: Initialized, Role: FOLLOWER
I20260620 02:27:08.538672 32475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33549 } }
I20260620 02:27:08.538892 32475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:27:08.539017 32475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:27:08.539224 32475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:08.541723 32475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33549 } }
I20260620 02:27:08.542363 32475 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3dfd8ea7944196ab5f49b23d4ca8b8; no voters:
I20260620 02:27:08.543015 32475 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:27:08.543236 32485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:08.544041 32485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3dfd8ea7944196ab5f49b23d4ca8b8, State: Running, Role: LEADER
I20260620 02:27:08.544621 32485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33549 } }
I20260620 02:27:08.545502 32475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:08.553246 32486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33549 } } }
I20260620 02:27:08.553923 32486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:08.554461 32487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3dfd8ea7944196ab5f49b23d4ca8b8. Latest consensus state: current_term: 1 leader_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3dfd8ea7944196ab5f49b23d4ca8b8" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 33549 } } }
I20260620 02:27:08.554929 32487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:08.560180 32499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:08.565599 32499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:27:08.574388 32499 catalog_manager.cc:1347] Generated new cluster ID: 1c51a515f733485ba0bf7063c59521eb
I20260620 02:27:08.574551 32499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:08.586973 32499 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:08.588593 32499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:08.596609 32499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3dfd8ea7944196ab5f49b23d4ca8b8: Generated new TSK 0
I20260620 02:27:08.597771 32499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:27:08.614892 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32408
W20260620 02:27:08.616463 32470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54595
W20260620 02:27:08.617306 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:27:08.617506 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 32408
2026-06-20T02:27:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:27:08 dist-test-slave-9vg1 krb5kdc[32506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:27:08 dist-test-slave-9vg1 krb5kdc[32506](info): set up 2 sockets
Jun 20 02:27:08 dist-test-slave-9vg1 krb5kdc[32506](info): commencing operation
krb5kdc: starting...
W20260620 02:27:09.757309 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:27:09 dist-test-slave-9vg1 krb5kdc[32506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:27:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:27:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:27:10.043677 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44377
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:10.396498 32519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:10.396899 32519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:10.397087 32519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:10.404807 32519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:10.405021 32519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:10.405143 32519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:10.405258 32519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:10.405349 32519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:10.416146 32519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:10.417645 32519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:10.419018 32519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:10.431421 32527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:10.431473 32524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:10.431389 32525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:10.432631 32519 server_base.cc:1034] running on GCE node
I20260620 02:27:10.433539 32519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:10.435207 32519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:10.436475 32519 hybrid_clock.cc:648] HybridClock initialized: now 1781922430436417 us; error 50 us; skew 500 ppm
I20260620 02:27:10.436880 32519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:10 dist-test-slave-9vg1 krb5kdc[32506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:10.440371 32519 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:10.442994 32519 webserver.cc:460] Webserver started at http://127.9.231.126:37957/ using document root <none> and password file <none>
I20260620 02:27:10.443783 32519 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:10.443972 32519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:10.444314 32519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:10.447537 32519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "ba15bb26ab2648628127406e614c5b66"
format_stamp: "Formatted at 2026-06-20 02:27:10 on dist-test-slave-9vg1"
server_key: "8053e8cbac8a2fb803a13c15b5fcb309"
server_key_iv: "be09ff38864635ac2242755eb2af0f1c"
server_key_version: "encryptionkey@0"
I20260620 02:27:10.448367 32519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "ba15bb26ab2648628127406e614c5b66"
format_stamp: "Formatted at 2026-06-20 02:27:10 on dist-test-slave-9vg1"
server_key: "8053e8cbac8a2fb803a13c15b5fcb309"
server_key_iv: "be09ff38864635ac2242755eb2af0f1c"
server_key_version: "encryptionkey@0"
I20260620 02:27:10.454236 32519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:27:10.458379 32534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:10.460031 32519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:27:10.460281 32519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "ba15bb26ab2648628127406e614c5b66"
format_stamp: "Formatted at 2026-06-20 02:27:10 on dist-test-slave-9vg1"
server_key: "8053e8cbac8a2fb803a13c15b5fcb309"
server_key_iv: "be09ff38864635ac2242755eb2af0f1c"
server_key_version: "encryptionkey@0"
I20260620 02:27:10.460516 32519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:10.466691 32519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:10.471535 32519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:10.471863 32519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:10.491775 32519 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:44377
I20260620 02:27:10.491828 32585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:44377 every 8 connection(s)
I20260620 02:27:10.493327 32519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:10.497205 32586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:10.502427 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 32519
I20260620 02:27:10.502652 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:10.503281 10141 external_mini_cluster.cc:1428] Setting key aa79c2e186a00592298b163f9fd69923
I20260620 02:27:10.509078 32586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66: Bootstrap starting.
I20260620 02:27:10.513645 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:10.505551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54607 (local address 127.9.231.126:44377)
0620 02:27:10.506698 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:10.506720 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:10.506771 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:27:10.508833 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:10.508861 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:10.509299 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:10.509763 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:10.511149 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:10.511160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:10.511181 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:10.511214 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:10.511311 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:10.511344 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:10.511626 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:10.511780 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:27:10.512043 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":606,"threads_started":1}
I20260620 02:27:10.513806 32586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:10.515386 32586 log.cc:824] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:10.518533 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:10.519112 32586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66: No bootstrap required, opened a new log
I20260620 02:27:10.527163 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:10.524185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41119 (local address 127.9.231.126:44377)
0620 02:27:10.524391 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:10.524398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:10.524879 (+ 481us) server_negotiation.cc:407] Connection header received
0620 02:27:10.525057 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:10.525064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:10.525135 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:10.525263 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:10.526351 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:10.526358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:10.526364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:27:10.526387 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:10.526441 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:27:10.526467 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:10.526616 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:10.526742 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 02:27:10.526805 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:27:10.529464 32586 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15bb26ab2648628127406e614c5b66" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44377 } }
I20260620 02:27:10.529868 32586 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:27:10.529978 32586 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba15bb26ab2648628127406e614c5b66, State: Initialized, Role: FOLLOWER
I20260620 02:27:10.530680 32586 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15bb26ab2648628127406e614c5b66" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44377 } }
I20260620 02:27:10.530948 32586 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:27:10.531131 32586 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:27:10.531327 32586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:10.533742 32586 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15bb26ab2648628127406e614c5b66" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44377 } }
I20260620 02:27:10.534319 32586 leader_election.cc:302] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba15bb26ab2648628127406e614c5b66; no voters:
I20260620 02:27:10.535082 32586 leader_election.cc:288] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:27:10.535284 32596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:10.536288 32596 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [term 1 LEADER]: Becoming Leader. State: Replica: ba15bb26ab2648628127406e614c5b66, State: Running, Role: LEADER
I20260620 02:27:10.536875 32596 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15bb26ab2648628127406e614c5b66" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44377 } }
I20260620 02:27:10.537467 32586 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:10.539906 32598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba15bb26ab2648628127406e614c5b66. Latest consensus state: current_term: 1 leader_uuid: "ba15bb26ab2648628127406e614c5b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15bb26ab2648628127406e614c5b66" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44377 } } }
I20260620 02:27:10.539834 32597 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba15bb26ab2648628127406e614c5b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15bb26ab2648628127406e614c5b66" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 44377 } } }
I20260620 02:27:10.540341 32598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:10.540436 32597 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:10.544498 32603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:10.550422 32603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:27:10.560976 32603 catalog_manager.cc:1347] Generated new cluster ID: 4bf8cb4d6e9f49e1b98ba84608772bd8
I20260620 02:27:10.561128 32603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:10.591180 32603 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:10.591776 32603 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:10.635226 32603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba15bb26ab2648628127406e614c5b66: Generated new TSK 0
I20260620 02:27:10.636216 32603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:27:10.644209 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32519
W20260620 02:27:10.645632 32581 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54607
W20260620 02:27:10.646576 10141 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32519: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:27:10.646756 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 32519
2026-06-20T02:27:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2030 ms)
[----------] 18 tests from AuthTokenIssuingTest (37294 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:27:10 dist-test-slave-9vg1 krb5kdc[32617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:27:10 dist-test-slave-9vg1 krb5kdc[32617](info): set up 2 sockets
Jun 20 02:27:10 dist-test-slave-9vg1 krb5kdc[32617](info): commencing operation
krb5kdc: starting...
W20260620 02:27:11.784826 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:27:11 dist-test-slave-9vg1 krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:27:12.071311 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:41407
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:41407,127.9.231.125:43965,127.9.231.124:41853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:12.421831 32630 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:27:12.422245 32630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:12.422346 32630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:12.422530 32630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:12.430210 32630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:12.430323 32630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:12.430393 32630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:12.430522 32630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:12.430644 32630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:12.441056 32630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:41407,127.9.231.125:43965,127.9.231.124:41853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:41407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:12.442466 32630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:12.443975 32630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:12.456204 32635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:12.456209 32636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:12.456457 32638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:12.457368 32630 server_base.cc:1034] running on GCE node
I20260620 02:27:12.458621 32630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:12.460213 32630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:12.461490 32630 hybrid_clock.cc:648] HybridClock initialized: now 1781922432461404 us; error 51 us; skew 500 ppm
I20260620 02:27:12.461865 32630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:12 dist-test-slave-9vg1 krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922432, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:12.471007 32630 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:12.472906 32630 webserver.cc:460] Webserver started at http://127.9.231.126:37987/ using document root <none> and password file <none>
I20260620 02:27:12.473738 32630 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:12.473884 32630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:12.474260 32630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:12.477592 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "6ab503ce498549dbab07c739960f7e61"
format_stamp: "Formatted at 2026-06-20 02:27:12 on dist-test-slave-9vg1"
server_key: "b669c8c981eef26dbc60d0dcf56d36d9"
server_key_iv: "eeb1b679b60c984b9e7ed3e724548351"
server_key_version: "encryptionkey@0"
I20260620 02:27:12.478431 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "6ab503ce498549dbab07c739960f7e61"
format_stamp: "Formatted at 2026-06-20 02:27:12 on dist-test-slave-9vg1"
server_key: "b669c8c981eef26dbc60d0dcf56d36d9"
server_key_iv: "eeb1b679b60c984b9e7ed3e724548351"
server_key_version: "encryptionkey@0"
I20260620 02:27:12.484138 32630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:27:12.488402 32645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:12.490151 32630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:27:12.490372 32630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "6ab503ce498549dbab07c739960f7e61"
format_stamp: "Formatted at 2026-06-20 02:27:12 on dist-test-slave-9vg1"
server_key: "b669c8c981eef26dbc60d0dcf56d36d9"
server_key_iv: "eeb1b679b60c984b9e7ed3e724548351"
server_key_version: "encryptionkey@0"
I20260620 02:27:12.490636 32630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:12.520736 32630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:12.525882 32630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:12.526147 32630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:12.548159 32630 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:41407
I20260620 02:27:12.548166 32696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:41407 every 8 connection(s)
I20260620 02:27:12.549667 32630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:12.550490 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 32630
I20260620 02:27:12.550755 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:12.551546 10141 external_mini_cluster.cc:1428] Setting key 9c43e2e3abc4d847964afaf6df471cf3
I20260620 02:27:12.554149 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:12.562299 32697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.231.125@KRBTEST.COM; defaulting to no policy
Jun 20 02:27:12 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922432, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.125@KRBTEST.COM" created.
I20260620 02:27:12.587014 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.566880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.9.231.126:41407)
0620 02:27:12.568696 (+ 1816us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:12.568705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:12.568760 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:27:12.570397 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:12.570404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:12.570642 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:12.570851 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:12.572257 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:12.573469 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:12.574885 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:12.575533 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:12.575736 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:12.579422 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:12.579430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:12.579443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:12.579473 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:12.582392 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:12.583161 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:12.583168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:12.583170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:12.583312 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:12.583721 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:12.583728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:12.583735 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:12.584265 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:12.584484 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:12.585056 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 02:27:12.585446 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":1574,"spinlock_wait_cycles":48512,"thread_start_us":1347,"threads_started":1}
I20260620 02:27:12.587049 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.566880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:41407 (local address 127.0.0.1:58884)
0620 02:27:12.567856 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.567904 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:12.570000 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:12.570874 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:12.570881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:12.571688 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:12.572029 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:12.572048 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:12.573636 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:12.573647 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:12.574726 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:12.574738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:12.575777 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:12.575783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:12.575961 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:12.576736 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:12.576760 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:12.579175 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:12.582541 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:12.582555 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:12.582569 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:12.582966 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:12.583428 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:12.583434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:12.583437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:12.583606 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:12.584404 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:12.584417 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:12.584655 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 02:27:12.584910 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 02:27:12.585706 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":241,"thread_start_us":441,"threads_started":1}
I20260620 02:27:12.594209 32697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:27:12.595460 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.594966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41870)
0620 02:27:12.595198 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.595331 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260620 02:27:12.595788 32647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
I20260620 02:27:12.597400 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.597057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41884)
0620 02:27:12.597203 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.597270 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:27:12.597618 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:43965: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab.
Entry for principal kudu/127.9.231.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab.
I20260620 02:27:12.619825 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:43965
--webserver_interface=127.9.231.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab
--principal=kudu/127.9.231.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:41407,127.9.231.125:43965,127.9.231.124:41853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:27:12.647037 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } attempt: 1
I20260620 02:27:12.648340 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.647830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41898)
0620 02:27:12.648037 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.648195 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 02:27:12.649570 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.649132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41904)
0620 02:27:12.649282 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.649382 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:27:12.650300 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:43965: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
I20260620 02:27:12.718647 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } attempt: 2
I20260620 02:27:12.720077 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.719461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41916)
0620 02:27:12.719761 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.719877 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 02:27:12.721258 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.720816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41920)
0620 02:27:12.720982 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.721096 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:27:12.721464 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:43965: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
I20260620 02:27:12.812789 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } attempt: 3
I20260620 02:27:12.814114 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.813614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41932)
0620 02:27:12.813864 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.813968 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 02:27:12.815383 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.814921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41934)
0620 02:27:12.815153 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.815230 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:27:12.815708 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:43965: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
I20260620 02:27:12.959021 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } attempt: 4
I20260620 02:27:12.960453 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.959800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41940)
0620 02:27:12.960160 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.960294 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 02:27:12.961505 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:12.961157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41948)
0620 02:27:12.961300 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:12.961387 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 02:27:12.961707 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:43965: Network error: Client connection negotiation failed: client connection to 127.9.231.125:43965: connect: Connection refused (error 111)
W20260620 02:27:12.973192 32704 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:27:12.973601 32704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:12.973701 32704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:12.973875 32704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:12.981392 32704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:12.981488 32704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:12.981551 32704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:12.981602 32704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:12.981658 32704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:12.992941 32704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:41407,127.9.231.125:43965,127.9.231.124:41853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab
--principal=kudu/127.9.231.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.231.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:12.994242 32704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:12.995738 32704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:13.008126 32709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:13.008154 32712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:13.008109 32710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:13.008872 32704 server_base.cc:1034] running on GCE node
I20260620 02:27:13.009673 32704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:13.011224 32704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:13.012480 32704 hybrid_clock.cc:648] HybridClock initialized: now 1781922433012418 us; error 41 us; skew 500 ppm
I20260620 02:27:13.012835 32704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:13.016191 32704 init.cc:377] Logged in from keytab as kudu/127.9.231.125@KRBTEST.COM (short username kudu)
I20260620 02:27:13.017779 32704 webserver.cc:460] Webserver started at http://127.9.231.125:34293/ using document root <none> and password file <none>
I20260620 02:27:13.018445 32704 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:13.018540 32704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:13.018816 32704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:13.021924 32704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data/instance:
uuid: "29214f186f91455caa3a9999b6710b38"
format_stamp: "Formatted at 2026-06-20 02:27:13 on dist-test-slave-9vg1"
server_key: "54c79b369da687e089a747020144e7e8"
server_key_iv: "5862d595acbb971d360482324a74a904"
server_key_version: "encryptionkey@0"
I20260620 02:27:13.022715 32704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/wal/instance:
uuid: "29214f186f91455caa3a9999b6710b38"
format_stamp: "Formatted at 2026-06-20 02:27:13 on dist-test-slave-9vg1"
server_key: "54c79b369da687e089a747020144e7e8"
server_key_iv: "5862d595acbb971d360482324a74a904"
server_key_version: "encryptionkey@0"
I20260620 02:27:13.028169 32704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:27:13.031733 32719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:13.033231 32704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:27:13.033437 32704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/wal
uuid: "29214f186f91455caa3a9999b6710b38"
format_stamp: "Formatted at 2026-06-20 02:27:13 on dist-test-slave-9vg1"
server_key: "54c79b369da687e089a747020144e7e8"
server_key_iv: "5862d595acbb971d360482324a74a904"
server_key_version: "encryptionkey@0"
I20260620 02:27:13.033679 32704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:13.053284 32704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:13.057976 32704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:13.058265 32704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:13.077502 32704 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.125:43965
I20260620 02:27:13.077519 302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.125:43965 every 8 connection(s)
I20260620 02:27:13.078999 32704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
I20260620 02:27:13.082842 303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:13.088165 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 32704
I20260620 02:27:13.088383 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-1/wal/instance
I20260620 02:27:13.088878 10141 external_mini_cluster.cc:1428] Setting key 7eedb11cb78cadcaa38d6d282b6ecdc2
I20260620 02:27:13.089668 303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:27:13.112695 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.092375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.9.231.126:41407)
0620 02:27:13.092862 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.092867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.093972 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 02:27:13.096236 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.096242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.096344 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.096470 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.098215 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.099164 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.100699 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.101428 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.101589 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.105344 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.105353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.105361 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.105421 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.109168 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.109962 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.109970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.109977 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.110149 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.110552 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.110561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.110567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.110996 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.111193 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.111918 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.112194 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":283,"threads_started":1}
I20260620 02:27:13.113456 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.093034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:41407 (local address 127.0.0.1:58888)
0620 02:27:13.093798 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.093850 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.095886 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.096724 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.096733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.097308 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.097963 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.097994 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.099323 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.099327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.100542 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.100554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.101664 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.101669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.101874 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.102587 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.102611 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.105075 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.109349 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.109357 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.109371 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.109824 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.110261 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.110265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.110267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.110438 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.111170 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.111177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.111536 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.111768 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.112274 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
I20260620 02:27:13.115465 303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.231.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.124@KRBTEST.COM" created.
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:27:13.133204 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.116628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.9.231.125:43965)
0620 02:27:13.117333 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.117339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.117364 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:27:13.117410 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.117415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.117610 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.117789 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.118666 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.120227 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.121466 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.122459 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.122618 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.125822 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.125843 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.125867 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.125917 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.129718 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.130779 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.130786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.130788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.130901 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.131408 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.131412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.131414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.131779 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.131955 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.132487 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.132895 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":557,"thread_start_us":391,"threads_started":1}
I20260620 02:27:13.133453 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.116179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41956)
0620 02:27:13.116351 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.116380 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.116473 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.117829 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.117833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.118226 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.118385 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.118394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.120406 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.120411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.121314 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.121328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.122612 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.122617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.122763 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.123491 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.123513 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.125584 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.129931 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.129940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.129947 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.130630 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.131045 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.131054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.131088 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.131291 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.131954 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.131964 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.132177 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.132372 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.132775 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 02:27:13.136270 303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:27:13.137552 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.137041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35370)
0620 02:27:13.137261 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.137395 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 02:27:13.137807 32721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
I20260620 02:27:13.139653 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.139296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35386)
0620 02:27:13.139454 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.139537 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:27:13.139843 303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab.
Entry for principal kudu/127.9.231.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab.
I20260620 02:27:13.158460 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:41853
--webserver_interface=127.9.231.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab
--principal=kudu/127.9.231.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:41407,127.9.231.125:43965,127.9.231.124:41853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:27:13.189133 303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 1
I20260620 02:27:13.190476 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.189797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35390)
0620 02:27:13.190068 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.190228 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 02:27:13.191761 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.191260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35402)
0620 02:27:13.191434 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.191574 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:27:13.192022 303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
I20260620 02:27:13.260253 303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 2
I20260620 02:27:13.260985 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } attempt: 5
I20260620 02:27:13.261516 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.260982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35418)
0620 02:27:13.261183 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.261292 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 02:27:13.263820 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.263386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35424)
0620 02:27:13.263568 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.263659 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 02:27:13.264074 303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922432, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:27:13.274765 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.261635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41970)
0620 02:27:13.261843 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.261863 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.261997 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.262390 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.262395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.262791 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.262963 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.262972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.264296 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.264301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.265189 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.265201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.266003 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.266010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.266209 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.266826 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.266844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.269178 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.272315 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.272321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.272325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.272697 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.273061 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.273066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.273070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.273208 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.273766 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.273771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.273944 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.274083 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.274405 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 02:27:13.274753 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.261911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.9.231.125:43965)
0620 02:27:13.262058 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.262064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.262085 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:27:13.262168 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.262175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.262242 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.262357 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.263148 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.264178 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.265344 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.265788 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.265933 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.269392 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.269399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.269403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.269443 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.272173 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.272823 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.272835 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.272837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.272948 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.273316 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.273322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.273325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.273600 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.273829 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.274209 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.274434 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:27:13.275816 32697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:27:13.276855 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.276463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35438)
0620 02:27:13.276636 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.276721 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 02:27:13.277776 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.277460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35446)
0620 02:27:13.277597 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.277679 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 02:27:13.278008 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
I20260620 02:27:13.329244 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 1
I20260620 02:27:13.330632 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.330037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35448)
0620 02:27:13.330288 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.330405 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 02:27:13.331750 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.331366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35460)
0620 02:27:13.331527 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.331615 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 02:27:13.332011 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
I20260620 02:27:13.355324 303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 3
I20260620 02:27:13.356743 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.356088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35462)
0620 02:27:13.356358 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.356503 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 02:27:13.358074 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.357547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35474)
0620 02:27:13.357804 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.357920 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260620 02:27:13.358681 303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
I20260620 02:27:13.400276 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 2
I20260620 02:27:13.401614 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.401012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35482)
0620 02:27:13.401280 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.401404 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 02:27:13.402976 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.402542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35496)
0620 02:27:13.402740 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.402838 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 02:27:13.403268 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
I20260620 02:27:13.501991 303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 4
I20260620 02:27:13.503464 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.502746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35498)
0620 02:27:13.503038 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.503228 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:27:13.504668 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.504260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35508)
0620 02:27:13.504462 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.504552 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 02:27:13.504945 303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
I20260620 02:27:13.509516 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 3
I20260620 02:27:13.510620 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.510149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35512)
0620 02:27:13.510344 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.510442 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:27:13.511735 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.511352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35514)
0620 02:27:13.511543 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.511624 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 02:27:13.511958 32697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:41853: Network error: Client connection negotiation failed: client connection to 127.9.231.124:41853: connect: Connection refused (error 111)
W20260620 02:27:13.516883 311 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:27:13.517270 311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:13.517376 311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:13.517513 311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:13.526415 311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:13.526535 311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:13.526657 311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:13.526751 311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:13.526841 311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:13.537569 311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:41407,127.9.231.125:43965,127.9.231.124:41853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab
--principal=kudu/127.9.231.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:41853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.231.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:13.539112 311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:13.540473 311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:13.554590 317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:13.554689 316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:13.555220 319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:13.555923 311 server_base.cc:1034] running on GCE node
I20260620 02:27:13.556782 311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:13.558431 311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:13.559711 311 hybrid_clock.cc:648] HybridClock initialized: now 1781922433559635 us; error 55 us; skew 500 ppm
I20260620 02:27:13.560113 311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:13.563512 311 init.cc:377] Logged in from keytab as kudu/127.9.231.124@KRBTEST.COM (short username kudu)
I20260620 02:27:13.565238 311 webserver.cc:460] Webserver started at http://127.9.231.124:38155/ using document root <none> and password file <none>
I20260620 02:27:13.566102 311 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:13.566252 311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:13.566601 311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:13.569777 311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data/instance:
uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740"
format_stamp: "Formatted at 2026-06-20 02:27:13 on dist-test-slave-9vg1"
server_key: "bc76de7bf371030e568512d0019e9915"
server_key_iv: "3c5d73b155684f191a87e6c592a2602f"
server_key_version: "encryptionkey@0"
I20260620 02:27:13.570655 311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/wal/instance:
uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740"
format_stamp: "Formatted at 2026-06-20 02:27:13 on dist-test-slave-9vg1"
server_key: "bc76de7bf371030e568512d0019e9915"
server_key_iv: "3c5d73b155684f191a87e6c592a2602f"
server_key_version: "encryptionkey@0"
I20260620 02:27:13.576300 311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 02:27:13.580313 326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:13.581832 311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:27:13.582050 311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/wal
uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740"
format_stamp: "Formatted at 2026-06-20 02:27:13 on dist-test-slave-9vg1"
server_key: "bc76de7bf371030e568512d0019e9915"
server_key_iv: "3c5d73b155684f191a87e6c592a2602f"
server_key_version: "encryptionkey@0"
I20260620 02:27:13.582330 311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:13.617697 311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:13.622349 311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:13.622668 311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:13.643296 311 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.124:41853
I20260620 02:27:13.643312 377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.124:41853 every 8 connection(s)
I20260620 02:27:13.644887 311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
I20260620 02:27:13.646955 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 311
I20260620 02:27:13.647174 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922055280103-10141-0/minicluster-data/master-2/wal/instance
I20260620 02:27:13.647657 10141 external_mini_cluster.cc:1428] Setting key 965cf451d95b29247caf38fa2bb4b33f
I20260620 02:27:13.649418 378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:13.659655 378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:27:13.673987 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.649861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.9.231.124:41853)
0620 02:27:13.650704 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.650716 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.650757 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:27:13.652645 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.652672 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.653153 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.653551 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.655611 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.657135 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.659010 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.659852 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.659993 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.665250 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.665258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.665271 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.665317 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.669056 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.670046 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.670062 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.670077 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.670307 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.670766 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.670773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.670776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.671513 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.671869 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.672360 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.672880 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":384,"threads_started":1}
I20260620 02:27:13.680346 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.662540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:41407 (local address 127.0.0.1:58904)
0620 02:27:13.663663 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.663744 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.663962 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.664844 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.664849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.665629 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.665928 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.665938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.667087 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.667095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.668989 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.669006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.670098 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.670104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.670308 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.671109 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.671141 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.673388 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.676941 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.676954 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.676962 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.677595 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.678170 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.678174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.678176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.678324 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.679004 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.679012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.679278 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.679492 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.679920 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":385,"threads_started":1}
I20260620 02:27:13.680342 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.662657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.9.231.126:41407)
0620 02:27:13.664436 (+ 1779us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.664444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.664469 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:27:13.664537 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.664542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.664629 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.664768 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.666101 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.666861 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.669177 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.669851 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.669996 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.673692 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.673697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.673701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.673746 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.676783 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.677767 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.677775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.677779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.677997 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.678549 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.678556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.678560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.678854 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.679019 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.679631 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.679891 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1631,"thread_start_us":1506,"threads_started":1}
I20260620 02:27:13.681804 378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:27:13.689196 32697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 4
I20260620 02:27:13.695662 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.677312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.9.231.126:41407)
0620 02:27:13.678055 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.678063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.678087 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:27:13.678135 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.678141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.678214 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.678332 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.679818 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.681173 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.682606 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.683296 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.683442 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.688166 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.688172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.688176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.688215 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.692266 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.693002 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.693010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.693014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.693205 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.693686 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.693691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.693694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.694048 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.694331 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.694902 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.695184 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":405,"threads_started":1}
I20260620 02:27:13.697428 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.682880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.9.231.125:43965)
0620 02:27:13.683101 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.683108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.683136 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:27:13.683194 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.683198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.683258 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.683336 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.684110 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.684781 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.685974 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.686516 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.686667 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.690406 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.690414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.690419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.690466 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.693980 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.694690 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.694697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.694700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.694860 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.695482 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.695487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.695490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.695873 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.696051 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.696729 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.697003 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:27:13.698486 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.682636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:43965 (local address 127.0.0.1:41982)
0620 02:27:13.682819 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.682838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.682938 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.683396 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.683402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.683813 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.683951 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.683960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.684922 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.684929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.685825 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.685837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.686725 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.686732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.686892 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.687633 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.687651 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.690147 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.694135 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.694140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.694144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.694546 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.695090 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.695096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.695100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.695340 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.696160 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.696166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.696420 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.697702 (+ 1282us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.698088 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922432, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:27:13.699622 378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:27:13.707180 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.690021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.9.231.124:41853)
0620 02:27:13.690205 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.690210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.690383 (+ 173us) server_negotiation.cc:407] Connection header received
0620 02:27:13.690491 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.690495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.690589 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.690737 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.691782 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.692993 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.694749 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.695473 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.696112 (+ 639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.699842 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.699850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.699859 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.699946 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.703938 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.704863 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.704868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.704872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.705027 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.705553 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.705564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.705567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.705934 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.706135 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.706526 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.706782 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":70}
I20260620 02:27:13.707194 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.690007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35532)
0620 02:27:13.690292 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.690317 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.690409 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.690795 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.690800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.691390 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.691621 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.691633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.693152 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.693163 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.694565 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.694578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.695659 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.695665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.695812 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.696416 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.696462 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.699576 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.704137 (+ 4561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.704145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.704151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.704696 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.705167 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.705174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.705179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.705395 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.706099 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.706105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.706273 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.706418 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.706748 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:27:13.714560 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61: Bootstrap starting.
I20260620 02:27:13.717815 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.698202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.9.231.125:43965)
0620 02:27:13.698361 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.698366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.698410 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:27:13.698853 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.698857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.698914 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.699023 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.700904 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.702050 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.704031 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.704550 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.704697 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.710021 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.710029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.710033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.710091 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.713776 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.714522 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.714530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.714535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.714716 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.715303 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.715309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.715312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.715663 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.715943 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.717198 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.717464 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:27:13.719223 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.700563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.9.231.124:41853)
0620 02:27:13.704464 (+ 3901us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.704472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.704494 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:27:13.704543 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.704549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.704637 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.704742 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.705785 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.707318 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.708959 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.709563 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.709692 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.713054 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.713059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.713063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.713092 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.716421 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.717065 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.717070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.717073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.717192 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.717554 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.717560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.717564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.717855 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.718052 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.718498 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.718748 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":3768,"thread_start_us":3554,"threads_started":1}
I20260620 02:27:13.719604 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.700368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35546)
0620 02:27:13.700665 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.700686 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.700785 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.705000 (+ 4215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.705006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.705395 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.705594 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.705604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.707473 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.707480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.708782 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.708795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.709748 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.709754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.709977 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.710561 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.710577 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.712830 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.716566 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.716572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.716575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.716934 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.717309 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.717314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.717318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.717445 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.718106 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.718109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.718250 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.718906 (+ 656us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.719287 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260620 02:27:13.719625 32697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:13.721143 32697 log.cc:824] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:13.723649 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:13.724643 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61: No bootstrap required, opened a new log
I20260620 02:27:13.725137 378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: Bootstrap starting.
I20260620 02:27:13.728300 378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:13.727182 32672 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ab503ce498549dbab07c739960f7e61"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58916
I20260620 02:27:13.729460 378 log.cc:824] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:13.732525 378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: No bootstrap required, opened a new log
I20260620 02:27:13.735208 32697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
I20260620 02:27:13.735641 32697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab503ce498549dbab07c739960f7e61, State: Initialized, Role: FOLLOWER
I20260620 02:27:13.736197 32697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
I20260620 02:27:13.737236 388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.737771 388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:13.738685 32697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:13.739009 32672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:27:13.739246 32672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:13.741384 378 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
I20260620 02:27:13.741912 378 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e5ad99c79549c1b0d69fc9cc8fc740, State: Initialized, Role: FOLLOWER
I20260620 02:27:13.742419 378 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
I20260620 02:27:13.743448 395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.743927 395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:13.743976 32672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
I20260620 02:27:13.745075 378 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:13.746078 32672 leader_election.cc:288] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [CANDIDATE]: Term 1 election: Requested vote from peers 29214f186f91455caa3a9999b6710b38 (127.9.231.125:43965), f8e5ad99c79549c1b0d69fc9cc8fc740 (127.9.231.124:41853)
I20260620 02:27:13.747571 353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab503ce498549dbab07c739960f7e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740"
I20260620 02:27:13.748185 353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:13.753101 353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab503ce498549dbab07c739960f7e61 in term 1.
I20260620 02:27:13.754283 32647 leader_election.cc:302] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ab503ce498549dbab07c739960f7e61, f8e5ad99c79549c1b0d69fc9cc8fc740; no voters:
I20260620 02:27:13.754989 388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:13.762503 388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab503ce498549dbab07c739960f7e61, State: Running, Role: LEADER
I20260620 02:27:13.763533 388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
W20260620 02:27:13.779309 32651 tablet.cc:2369] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:27:13.787415 402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab503ce498549dbab07c739960f7e61. Latest consensus state: current_term: 1 leader_uuid: "6ab503ce498549dbab07c739960f7e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.787887 402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:13.789484 412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:13.807121 303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } attempt: 5
I20260620 02:27:13.811198 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.777680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.9.231.125:43965)
0620 02:27:13.777902 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.777907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.781448 (+ 3541us) server_negotiation.cc:407] Connection header received
0620 02:27:13.783076 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.783083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.783202 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.783283 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.791567 (+ 8284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.793074 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.798611 (+ 5537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.799300 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.799418 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.802703 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.802710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.802714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.802763 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.806573 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.807651 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.807656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.807659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.807834 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.808303 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.808309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.808313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.808734 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.808965 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.809592 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.810628 (+ 1036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:27:13.816437 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.782083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.9.231.124:41853)
0620 02:27:13.782275 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.782282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.783415 (+ 1133us) server_negotiation.cc:407] Connection header received
0620 02:27:13.783830 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.783837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.783928 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.784005 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.795546 (+ 11541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.796660 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.804095 (+ 7435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.804865 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.804980 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.808676 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.808683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.808688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.808729 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.812503 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.813923 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.813931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.813935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.814100 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.814543 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.814548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.814551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.814943 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.815201 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.815750 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.816004 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 02:27:13 dist-test-slave-9vg1 krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922433, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:27:13.823251 412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:27:13.803315 32630 master_runner.cc:186] Error getting master registration for 127.9.231.125:43965: OK, instance_id { permanent_uuid: "29214f186f91455caa3a9999b6710b38" instance_seqno: 1781922433071941 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:27:13.829785 32630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:27:13.831401 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.771445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.9.231.126:41407)
0620 02:27:13.771782 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.771792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.776981 (+ 5189us) server_negotiation.cc:407] Connection header received
0620 02:27:13.777959 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.777966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.778053 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.778129 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.780537 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.781741 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.795600 (+ 13859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.796334 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.796443 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.801384 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.801391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.801397 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.801454 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.814914 (+ 13460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.824861 (+ 9947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.824871 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.824875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.825091 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.825970 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.825975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.825978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.826392 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.826713 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.830402 (+ 3689us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.830798 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 02:27:13.832599 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.808642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.9.231.124:41853)
0620 02:27:13.808895 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:13.808904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:13.808928 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:27:13.808999 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:13.809005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:13.809102 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:13.809216 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:13.810546 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.811952 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.814102 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.814783 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.814905 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.820593 (+ 5688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:13.820602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:13.820609 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:13.820648 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:13.825650 (+ 5002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.827104 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.827111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.827115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.827370 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:13.828106 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:13.828114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:13.828117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:13.828841 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:13.829545 (+ 704us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:13.830096 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 02:27:13.830998 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:27:13.832667 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:13.807779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:41853 (local address 127.0.0.1:35552)
0620 02:27:13.808229 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:13.808251 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:13.808377 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:13.809438 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:13.809446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:13.810083 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:13.810341 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.810358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.812168 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.812174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.813604 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:13.813623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:13.815189 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:13.815195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:13.815575 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:13.816416 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:13.816464 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:13.820276 (+ 3812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:13.825864 (+ 5588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.825871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.825875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.826862 (+ 987us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.827633 (+ 771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:13.827638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:13.827642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:13.827910 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:13.829072 (+ 1162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:13.829079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:13.829494 (+ 415us) client_negotiation.cc:769] Sending connection context
0620 02:27:13.830159 (+ 665us) client_negotiation.cc:240] Negotiation successful
0620 02:27:13.831258 (+ 1099us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290}
I20260620 02:27:13.838939 303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38: Bootstrap starting.
I20260620 02:27:13.844828 303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:13.847383 303 log.cc:824] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:13.856284 303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38: No bootstrap required, opened a new log
I20260620 02:27:13.871160 353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [term 1 FOLLOWER]: Refusing update from remote peer 6ab503ce498549dbab07c739960f7e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:27:13.872273 388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:27:13.875154 332 tablet.cc:2369] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:27:13.881634 303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
I20260620 02:27:13.882117 303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29214f186f91455caa3a9999b6710b38, State: Initialized, Role: FOLLOWER
I20260620 02:27:13.884909 303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } }
I20260620 02:27:13.888232 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.888689 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:13.890650 303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:13.890733 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab503ce498549dbab07c739960f7e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29214f186f91455caa3a9999b6710b38"
I20260620 02:27:13.891460 32746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:13.896092 395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab503ce498549dbab07c739960f7e61. Latest consensus state: current_term: 1 leader_uuid: "6ab503ce498549dbab07c739960f7e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.896433 395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:13.899891 32746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab503ce498549dbab07c739960f7e61 in term 1.
I20260620 02:27:13.900034 417 mvcc.cc:205] Tried to move back new op lower bound from 7298754289122414592 to 7298754288704147456. Current Snapshot: MvccSnapshot[applied={T|T < 7298754289122414592}]
I20260620 02:27:13.900957 32745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [term 1 FOLLOWER]: Refusing update from remote peer 6ab503ce498549dbab07c739960f7e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:27:13.902990 402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:27:13.911257 422 mvcc.cc:205] Tried to move back new op lower bound from 7298754289122414592 to 7298754288704147456. Current Snapshot: MvccSnapshot[applied={T|T < 7298754289122414592}]
I20260620 02:27:13.913162 388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ab503ce498549dbab07c739960f7e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.913478 388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:13.915881 412 catalog_manager.cc:1347] Generated new cluster ID: a5e0d7823c9741d48b7c3127ff052338
I20260620 02:27:13.916016 412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:13.937177 395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ab503ce498549dbab07c739960f7e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.937476 395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:13.943156 433 mvcc.cc:205] Tried to move back new op lower bound from 7298754289122414592 to 7298754288704147456. Current Snapshot: MvccSnapshot[applied={T|T < 7298754289122414592}]
I20260620 02:27:13.948979 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab503ce498549dbab07c739960f7e61. Latest consensus state: current_term: 1 leader_uuid: "6ab503ce498549dbab07c739960f7e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.949880 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:13.950470 437 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:27:13.949473 435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ab503ce498549dbab07c739960f7e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.951593 435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:13.953296 388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ab503ce498549dbab07c739960f7e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab503ce498549dbab07c739960f7e61" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 41407 } } peers { permanent_uuid: "29214f186f91455caa3a9999b6710b38" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 43965 } } peers { permanent_uuid: "f8e5ad99c79549c1b0d69fc9cc8fc740" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 41853 } } }
I20260620 02:27:13.953563 388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:13.959306 412 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:13.961194 412 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:13.980398 412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ab503ce498549dbab07c739960f7e61: Generated new TSK 0
I20260620 02:27:13.981848 412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:27:13.967779 311 master_runner.cc:186] Error getting master registration for 127.9.231.125:43965: OK, instance_id { permanent_uuid: "29214f186f91455caa3a9999b6710b38" instance_seqno: 1781922433071941 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:27:13.990679 311 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:27:14.012934 446 catalog_manager.cc:1259] Loaded cluster ID: a5e0d7823c9741d48b7c3127ff052338
I20260620 02:27:14.013130 446 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38: loading cluster ID for follower catalog manager: success
I20260620 02:27:14.017619 446 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38: acquiring CA information for follower catalog manager: success
I20260620 02:27:14.020761 446 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29214f186f91455caa3a9999b6710b38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:27:14.079159 384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.076303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.9.231.126:41407)
0620 02:27:14.076510 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.076517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.077063 (+ 546us) server_negotiation.cc:407] Connection header received
0620 02:27:14.077355 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.077362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.077453 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.077535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.078841 (+ 1306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58934: BlockingRecv error: recv got EOF from 127.0.0.1:58934 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 02:27:14.082532 308 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.079387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.9.231.125:43965)
0620 02:27:14.079948 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.079954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.080815 (+ 861us) server_negotiation.cc:407] Connection header received
0620 02:27:14.081076 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.081082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.081184 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.081258 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.082291 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42002: BlockingRecv error: recv got EOF from 127.0.0.1:42002 (error 108)
Metrics: {"server-negotiator.queue_time_us":380}
W20260620 02:27:14.097750 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.095557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.9.231.124:41853)
0620 02:27:14.095791 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.095797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.095824 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:27:14.095978 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.095984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.096070 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.096189 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.097486 (+ 1297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35562: BlockingRecv error: recv got EOF from 127.0.0.1:35562 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:27:14.139225 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.128363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.9.231.125:43965)
0620 02:27:14.128689 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.128701 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.131842 (+ 3141us) server_negotiation.cc:407] Connection header received
0620 02:27:14.132231 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.132236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.132447 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.132673 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.133311 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.134223 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.135973 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.136524 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.136696 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.137352 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.138101 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.138312 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.138586 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.138745 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260620 02:27:14.151912 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.149033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.9.231.124:41853)
0620 02:27:14.149320 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.149331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.149979 (+ 648us) server_negotiation.cc:407] Connection header received
0620 02:27:14.150303 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.150309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.150462 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.150610 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.151616 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35578: BlockingRecv error: recv got EOF from 127.0.0.1:35578 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 02:27:14.162942 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.156832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.9.231.125:43965)
0620 02:27:14.157095 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.157104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.157670 (+ 566us) server_negotiation.cc:407] Connection header received
0620 02:27:14.157914 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.157918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.158005 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.158109 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.158546 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.159442 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.160999 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.161494 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.161659 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.162102 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.162234 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.162343 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.162511 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.162601 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 02:27:14.181396 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.172508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.9.231.124:41853)
0620 02:27:14.172761 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.172771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.179681 (+ 6910us) server_negotiation.cc:407] Connection header received
0620 02:27:14.179909 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.179916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.180024 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.180155 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.181121 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35580: BlockingRecv error: recv got EOF from 127.0.0.1:35580 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:27:14.200644 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.193359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.9.231.125:43965)
0620 02:27:14.193602 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.193610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.194176 (+ 566us) server_negotiation.cc:407] Connection header received
0620 02:27:14.194387 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.194394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.194495 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.194620 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.195180 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.196424 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.198396 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.199177 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.199318 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.199770 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.199927 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.200018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.200170 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.200267 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260620 02:27:14.215616 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.213072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.9.231.124:41853)
0620 02:27:14.213282 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.213288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.213915 (+ 627us) server_negotiation.cc:407] Connection header received
0620 02:27:14.214126 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.214132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.214241 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.214357 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.215347 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35582: BlockingRecv error: recv got EOF from 127.0.0.1:35582 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:27:14.232342 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.224923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.9.231.125:43965)
0620 02:27:14.225170 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.225177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.225726 (+ 549us) server_negotiation.cc:407] Connection header received
0620 02:27:14.225970 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.225976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.226076 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.226181 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.226708 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.227945 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.230013 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.230772 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.230914 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.231501 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.231665 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.231762 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.231927 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.232028 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260620 02:27:14.248879 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.246117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.9.231.124:41853)
0620 02:27:14.246449 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.246456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.247149 (+ 693us) server_negotiation.cc:407] Connection header received
0620 02:27:14.247391 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.247397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.247523 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.247643 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.248628 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35586: BlockingRecv error: recv got EOF from 127.0.0.1:35586 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
I20260620 02:27:14.272747 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.265285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.9.231.125:43965)
0620 02:27:14.265489 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.265496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.266127 (+ 631us) server_negotiation.cc:407] Connection header received
0620 02:27:14.266341 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.266349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.266461 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.266579 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.267089 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.268203 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.270325 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.271093 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.271234 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.271777 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.271968 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.272068 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.272249 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.272360 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 02:27:14.282226 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.280360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.9.231.124:41853)
0620 02:27:14.280540 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.280547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.280568 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:27:14.280861 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.280867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.280959 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.281057 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.281965 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35602: BlockingRecv error: recv got EOF from 127.0.0.1:35602 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:27:14.310395 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.303182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.9.231.125:43965)
0620 02:27:14.303394 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.303400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.303965 (+ 565us) server_negotiation.cc:407] Connection header received
0620 02:27:14.304195 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.304201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.304291 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.304393 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.304917 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.306134 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.308176 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.308837 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.308980 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.309501 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.309696 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.309794 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.309972 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.310070 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 02:27:14.320433 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.318091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.9.231.124:41853)
0620 02:27:14.318289 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.318295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.318839 (+ 544us) server_negotiation.cc:407] Connection header received
0620 02:27:14.319044 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.319051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.319175 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.319293 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.320185 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35604: BlockingRecv error: recv got EOF from 127.0.0.1:35604 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:27:14.365010 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.357205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.9.231.125:43965)
0620 02:27:14.357459 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.357466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.358125 (+ 659us) server_negotiation.cc:407] Connection header received
0620 02:27:14.358314 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.358320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.358451 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.358577 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.359185 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.360417 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.362566 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.363379 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.363539 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.364070 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.364229 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.364345 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.364527 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.364633 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 02:27:14.383632 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.381017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.9.231.124:41853)
0620 02:27:14.381215 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.381222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.381846 (+ 624us) server_negotiation.cc:407] Connection header received
0620 02:27:14.382062 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.382069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.382174 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.382302 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.383355 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35618: BlockingRecv error: recv got EOF from 127.0.0.1:35618 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:27:14.457557 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.451190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.9.231.125:43965)
0620 02:27:14.451372 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.451378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.451755 (+ 377us) server_negotiation.cc:407] Connection header received
0620 02:27:14.451960 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.451977 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.452084 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.452195 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.452624 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.453830 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.455354 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.456051 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.456207 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.456705 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.456857 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.456944 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.457141 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.457237 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 02:27:14.476926 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.474780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.9.231.124:41853)
0620 02:27:14.475045 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.475070 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.475348 (+ 278us) server_negotiation.cc:407] Connection header received
0620 02:27:14.475601 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.475605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.475683 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.475800 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.476684 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35624: BlockingRecv error: recv got EOF from 127.0.0.1:35624 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:27:14.613822 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.608693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.9.231.125:43965)
0620 02:27:14.608938 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.608945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.609112 (+ 167us) server_negotiation.cc:407] Connection header received
0620 02:27:14.609391 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.609396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.609527 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.609635 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.610108 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.610798 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.612130 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.612553 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.612684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.613097 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.613205 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.613302 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.613454 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.613535 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260620 02:27:14.622603 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.620536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.9.231.124:41853)
0620 02:27:14.620806 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.620814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.621081 (+ 267us) server_negotiation.cc:407] Connection header received
0620 02:27:14.621305 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.621313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.621414 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.621528 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.622346 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35626: BlockingRecv error: recv got EOF from 127.0.0.1:35626 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 02:27:14.887869 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:14.882401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.9.231.125:43965)
0620 02:27:14.882593 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.882598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.882873 (+ 275us) server_negotiation.cc:407] Connection header received
0620 02:27:14.883133 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.883139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.883230 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.883321 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:14.883717 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.884480 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.886087 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:14.886611 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:14.886738 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:14.887133 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:14.887266 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:14.887338 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:14.887521 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 02:27:14.887627 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 02:27:14.896531 381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:14.894475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.9.231.124:41853)
0620 02:27:14.894755 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:14.894760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:14.894993 (+ 233us) server_negotiation.cc:407] Connection header received
0620 02:27:14.895231 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:14.895236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:14.895314 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:14.895407 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:14.896255 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35632: BlockingRecv error: recv got EOF from 127.0.0.1:35632 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 02:27:14.955399 437 catalog_manager.cc:1259] Loaded cluster ID: a5e0d7823c9741d48b7c3127ff052338
I20260620 02:27:14.955627 437 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: loading cluster ID for follower catalog manager: success
I20260620 02:27:14.958791 437 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: acquiring CA information for follower catalog manager: success
I20260620 02:27:14.961079 437 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8e5ad99c79549c1b0d69fc9cc8fc740: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:27:15.416837 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:15.411697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.9.231.125:43965)
0620 02:27:15.412150 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:15.412155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:15.412239 (+ 84us) server_negotiation.cc:407] Connection header received
0620 02:27:15.412410 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:15.412414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:15.412490 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:15.412607 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:15.413018 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.413806 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.415250 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.415645 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.415738 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:15.416112 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:15.416217 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:15.416286 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:15.416490 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 02:27:15.416557 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":261,"threads_started":1}
I20260620 02:27:15.429536 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:15.423529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.9.231.124:41853)
0620 02:27:15.424012 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:15.424016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:15.424161 (+ 145us) server_negotiation.cc:407] Connection header received
0620 02:27:15.424333 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:15.424337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:15.424504 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:15.424649 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:15.425145 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.425832 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.427187 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.427652 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.427796 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:15.428337 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:15.428880 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:15.429059 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:15.429156 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 02:27:15.429239 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":287,"threads_started":1}
I20260620 02:27:15.444255 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:15.436839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.9.231.125:43965)
0620 02:27:15.437032 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:15.437040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:15.437638 (+ 598us) server_negotiation.cc:407] Connection header received
0620 02:27:15.437741 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:15.437746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:15.437835 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:15.437931 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:15.438747 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.439855 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.442051 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.442661 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.442784 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:15.443436 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:15.443558 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:15.443663 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:15.443838 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 02:27:15.443939 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:27:15.444753 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:15.435894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.9.231.126:41407)
0620 02:27:15.436431 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:15.436440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:15.436876 (+ 436us) server_negotiation.cc:407] Connection header received
0620 02:27:15.437161 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:15.437166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:15.437352 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:15.437535 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:15.438249 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.439389 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.441626 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.442349 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.442564 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:15.443209 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:15.443789 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:15.443900 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:15.444368 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 02:27:15.444463 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":287,"threads_started":1}
I20260620 02:27:15.445451 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:15.437543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.9.231.124:41853)
0620 02:27:15.438250 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:15.438256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:15.439151 (+ 895us) server_negotiation.cc:407] Connection header received
0620 02:27:15.439393 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:15.439398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:15.439509 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:15.439602 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:15.440350 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.441068 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.443359 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:15.443948 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:15.444078 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:15.444690 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:15.444819 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:15.444901 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:15.445090 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 02:27:15.445171 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571}
I20260620 02:27:15.450436 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32630
I20260620 02:27:15.451594 32692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:15.566218 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 32630
I20260620 02:27:15.584779 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 32704
I20260620 02:27:15.585970 32766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:15.697372 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 32704
I20260620 02:27:15.718513 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 311
I20260620 02:27:15.719640 373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:15.828454 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 311
2026-06-20T02:27:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5186 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:27:15 dist-test-slave-9vg1 krb5kdc[584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:27:15 dist-test-slave-9vg1 krb5kdc[584](info): set up 2 sockets
Jun 20 02:27:15 dist-test-slave-9vg1 krb5kdc[584](info): commencing operation
krb5kdc: starting...
W20260620 02:27:16.969120 10141 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/test-user.keytab.
Jun 20 02:27:17 dist-test-slave-9vg1 krb5kdc[584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:27:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.231.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
Entry for principal kudu/127.9.231.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab.
I20260620 02:27:17.259478 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38395
--webserver_interface=127.9.231.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38395,127.9.231.125:39197,127.9.231.124:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:27:17.624522 597 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:27:17.624944 597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:17.625061 597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:17.625198 597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:17.633144 597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:17.633272 597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:17.633338 597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:17.633455 597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:17.633571 597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:17.644322 597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38395,127.9.231.125:39197,127.9.231.124:38879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.126.keytab
--principal=kudu/127.9.231.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.126:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.231.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:17.645879 597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:17.647505 597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:17.660918 602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:17.660842 605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:17.660881 603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:17.661751 597 server_base.cc:1034] running on GCE node
I20260620 02:27:17.662614 597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:17.664218 597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:17.665485 597 hybrid_clock.cc:648] HybridClock initialized: now 1781922437665407 us; error 46 us; skew 500 ppm
I20260620 02:27:17.665889 597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:17 dist-test-slave-9vg1 krb5kdc[584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:17.674036 597 init.cc:377] Logged in from keytab as kudu/127.9.231.126@KRBTEST.COM (short username kudu)
I20260620 02:27:17.675865 597 webserver.cc:460] Webserver started at http://127.9.231.126:39215/ using document root <none> and password file <none>
I20260620 02:27:17.676625 597 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:17.676767 597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:17.677105 597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:17.680362 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data/instance:
uuid: "0a62608587c54c2bb244ee7ecd6e0642"
format_stamp: "Formatted at 2026-06-20 02:27:17 on dist-test-slave-9vg1"
server_key: "d787a7d364d8323f2390f0f579f65dec"
server_key_iv: "3cf63aed1257ddc2edff022a3072d103"
server_key_version: "encryptionkey@0"
I20260620 02:27:17.681227 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/wal/instance:
uuid: "0a62608587c54c2bb244ee7ecd6e0642"
format_stamp: "Formatted at 2026-06-20 02:27:17 on dist-test-slave-9vg1"
server_key: "d787a7d364d8323f2390f0f579f65dec"
server_key_iv: "3cf63aed1257ddc2edff022a3072d103"
server_key_version: "encryptionkey@0"
I20260620 02:27:17.687036 597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:27:17.691013 612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:17.693563 597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 02:27:17.693755 597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/wal
uuid: "0a62608587c54c2bb244ee7ecd6e0642"
format_stamp: "Formatted at 2026-06-20 02:27:17 on dist-test-slave-9vg1"
server_key: "d787a7d364d8323f2390f0f579f65dec"
server_key_iv: "3cf63aed1257ddc2edff022a3072d103"
server_key_version: "encryptionkey@0"
I20260620 02:27:17.693966 597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:17.710911 597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:17.715760 597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:17.716089 597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:17.736356 597 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.126:38395
I20260620 02:27:17.736378 664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.126:38395 every 8 connection(s)
I20260620 02:27:17.737884 597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/data/info.pb
I20260620 02:27:17.739491 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 597
I20260620 02:27:17.739691 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-0/wal/instance
I20260620 02:27:17.740257 10141 external_mini_cluster.cc:1428] Setting key fdad8df94ef2181509badadf53dc77c6
I20260620 02:27:17.743041 665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:17.753598 665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.231.125@KRBTEST.COM; defaulting to no policy
Jun 20 02:27:17 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.125@KRBTEST.COM" created.
I20260620 02:27:17.778183 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.758675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.9.231.126:38395)
0620 02:27:17.759559 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:17.759573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:17.759628 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:27:17.760873 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:17.760879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:17.761143 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:17.761364 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:17.762715 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:17.764076 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:17.765596 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:17.766187 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:17.766394 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:17.770293 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:17.770302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:17.770317 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:17.770354 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:17.773609 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:17.774625 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:17.774632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:17.774634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:17.774786 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:17.775313 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:17.775317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:17.775319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:17.775914 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:17.776095 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:17.776767 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 02:27:17.777041 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":591,"thread_start_us":399,"threads_started":1}
I20260620 02:27:17.778211 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.757261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38395 (local address 127.0.0.1:60928)
0620 02:27:17.758112 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:17.758150 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:17.760460 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:17.761383 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:17.761390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:17.762129 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:17.762487 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:17.762506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:17.764237 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:17.764242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:17.765454 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:17.765472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:17.766437 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:17.766454 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:17.766697 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:17.767434 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:17.767470 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:17.769976 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:17.773776 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:17.773794 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:17.773811 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:17.774431 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:17.774907 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:17.774915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:17.774921 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:17.775186 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:17.776064 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:17.776075 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:17.776336 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 02:27:17.776624 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 02:27:17.777217 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":75,"thread_start_us":361,"threads_started":1}
I20260620 02:27:17.781754 665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:27:17.783000 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.782599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43430)
0620 02:27:17.782774 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:17.782883 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 02:27:17.783458 613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
I20260620 02:27:17.785152 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.784785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43438)
0620 02:27:17.784940 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:17.785013 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:27:17.785449 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:39197: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab.
Entry for principal kudu/127.9.231.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab.
I20260620 02:27:17.807693 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:39197
--webserver_interface=127.9.231.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab
--principal=kudu/127.9.231.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38395,127.9.231.125:39197,127.9.231.124:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:27:17.834802 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } attempt: 1
I20260620 02:27:17.836227 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.835701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43446)
0620 02:27:17.835928 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:17.836086 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 02:27:17.837404 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.836996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43458)
0620 02:27:17.837167 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:17.837254 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 02:27:17.837711 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:39197: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
I20260620 02:27:17.906044 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } attempt: 2
I20260620 02:27:17.907394 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.906835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43472)
0620 02:27:17.907154 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:17.907247 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 02:27:17.908560 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:17.908167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43480)
0620 02:27:17.908336 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:17.908418 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:27:17.908844 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:39197: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
I20260620 02:27:18.000118 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } attempt: 3
I20260620 02:27:18.001348 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.000905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43494)
0620 02:27:18.001115 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.001210 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 02:27:18.002616 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.002177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43498)
0620 02:27:18.002362 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.002459 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:27:18.002830 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:39197: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
I20260620 02:27:18.146128 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } attempt: 4
I20260620 02:27:18.147517 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.146877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43508)
0620 02:27:18.147266 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.147359 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260620 02:27:18.148824 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.148341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43516)
0620 02:27:18.148591 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.148684 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 02:27:18.149029 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.125:39197: Network error: Client connection negotiation failed: client connection to 127.9.231.125:39197: connect: Connection refused (error 111)
W20260620 02:27:18.161854 672 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:27:18.162285 672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:18.162422 672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:18.162563 672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:18.170517 672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:18.170668 672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:18.170794 672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:18.170881 672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:18.170992 672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:18.181646 672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38395,127.9.231.125:39197,127.9.231.124:38879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.125.keytab
--principal=kudu/127.9.231.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.125:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.231.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:18.183176 672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:18.184726 672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:18.197860 677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:18.197861 678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:18.197911 680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:18.199290 672 server_base.cc:1034] running on GCE node
I20260620 02:27:18.200145 672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:18.201658 672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:18.202948 672 hybrid_clock.cc:648] HybridClock initialized: now 1781922438202862 us; error 63 us; skew 500 ppm
I20260620 02:27:18.203378 672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:18.206794 672 init.cc:377] Logged in from keytab as kudu/127.9.231.125@KRBTEST.COM (short username kudu)
I20260620 02:27:18.208552 672 webserver.cc:460] Webserver started at http://127.9.231.125:36971/ using document root <none> and password file <none>
I20260620 02:27:18.209317 672 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:18.209455 672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:18.209780 672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:18.212961 672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data/instance:
uuid: "2739e08f46fc4b9bb4b7760392b0d1b1"
format_stamp: "Formatted at 2026-06-20 02:27:18 on dist-test-slave-9vg1"
server_key: "1a59d7e2179e25f68ac67847b95f873d"
server_key_iv: "7008259b71f0b973d337a4f169f2bbc5"
server_key_version: "encryptionkey@0"
I20260620 02:27:18.213809 672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/wal/instance:
uuid: "2739e08f46fc4b9bb4b7760392b0d1b1"
format_stamp: "Formatted at 2026-06-20 02:27:18 on dist-test-slave-9vg1"
server_key: "1a59d7e2179e25f68ac67847b95f873d"
server_key_iv: "7008259b71f0b973d337a4f169f2bbc5"
server_key_version: "encryptionkey@0"
I20260620 02:27:18.219413 672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 02:27:18.223579 687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:18.224992 672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:27:18.225203 672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/wal
uuid: "2739e08f46fc4b9bb4b7760392b0d1b1"
format_stamp: "Formatted at 2026-06-20 02:27:18 on dist-test-slave-9vg1"
server_key: "1a59d7e2179e25f68ac67847b95f873d"
server_key_iv: "7008259b71f0b973d337a4f169f2bbc5"
server_key_version: "encryptionkey@0"
I20260620 02:27:18.225440 672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:18.243280 672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:18.248423 672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:18.248771 672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:18.268543 672 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.125:39197
I20260620 02:27:18.268550 738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.125:39197 every 8 connection(s)
I20260620 02:27:18.270037 672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/data/info.pb
I20260620 02:27:18.274062 739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:18.276787 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 672
I20260620 02:27:18.276963 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-1/wal/instance
I20260620 02:27:18.277494 10141 external_mini_cluster.cc:1428] Setting key 3073fdc83db40fdca0ec526d9375ad17
I20260620 02:27:18.283895 739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.231.124@KRBTEST.COM; defaulting to no policy
I20260620 02:27:18.307969 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.287798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.9.231.126:38395)
0620 02:27:18.288421 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.288430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.289472 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 02:27:18.291628 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.291635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.291746 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.291875 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.293703 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.294594 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.296300 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.296733 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.296882 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.300922 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.300928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.300931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.300981 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.304164 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.305177 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.305182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.305185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.305320 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.305800 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.305811 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.305813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.306128 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.306360 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.307201 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.307443 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":286,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.231.124@KRBTEST.COM" created.
I20260620 02:27:18.309098 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.288460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38395 (local address 127.0.0.1:60936)
0620 02:27:18.289282 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.289324 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:18.291288 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:18.292154 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:18.292163 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:18.292777 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:18.293448 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.293468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.294778 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.294783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.296134 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.296148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.296984 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.296991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.297315 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.298108 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:18.298133 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:18.300569 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:18.304366 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.304377 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.304399 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.304988 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.305448 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.305453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.305457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.305669 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.306335 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:18.306344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:18.306785 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 02:27:18.307072 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 02:27:18.307659 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":309,"threads_started":1}
I20260620 02:27:18.311260 739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:27:18.329357 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.312214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:43530)
0620 02:27:18.312427 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.312451 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:18.312574 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:18.313611 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:18.313615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:18.314105 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:18.314238 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.314247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.315769 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.315775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.316670 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.316681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.317557 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.317563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.317737 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.318457 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:18.318488 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:18.321216 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:18.326141 (+ 4925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.326145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.326148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.326646 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.327190 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.327194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.327196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.327357 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.328207 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:18.328216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:18.328357 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 02:27:18.328545 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 02:27:18.328871 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 02:27:18.329635 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.312475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.9.231.125:39197)
0620 02:27:18.313156 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.313162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.313188 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:27:18.313236 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.313241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.313455 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.313628 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.314423 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.315632 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.316813 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.317383 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.317537 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.321508 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.321530 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.321555 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.321602 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.325925 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.326813 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.326826 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.326832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.326985 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.327492 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.327499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.327505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.328039 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.328277 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.328725 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.329087 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":385,"threads_started":1}
I20260620 02:27:18.332360 739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:27:18.333705 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.333069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:53308)
0620 02:27:18.333322 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.333486 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 02:27:18.334026 688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.336023 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.335445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:53310)
0620 02:27:18.335641 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.335815 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 02:27:18.336341 739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.231.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab.
Entry for principal kudu/127.9.231.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab.
I20260620 02:27:18.347250 10141 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4D_H_y/build/asan/bin/kudu
/tmp/dist-test-task4D_H_y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:38879
--webserver_interface=127.9.231.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab
--principal=kudu/127.9.231.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.231.84:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.231.126:38395,127.9.231.125:39197,127.9.231.124:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:27:18.385746 739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 1
I20260620 02:27:18.387259 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.386483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:53322)
0620 02:27:18.386847 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.387018 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 02:27:18.388711 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.388264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:53330)
0620 02:27:18.388460 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.388561 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:27:18.388978 739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.448335 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } attempt: 5
I20260620 02:27:18.457216 739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 2
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:27:18.458415 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.457969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56738)
0620 02:27:18.458164 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.458257 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 02:27:18.459722 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.459254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56740)
0620 02:27:18.459424 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.459551 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:27:18.460016 739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.465317 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.449366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:44976)
0620 02:27:18.449648 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.449673 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:18.449881 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:18.450304 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:18.450308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:18.450800 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:18.450997 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.451007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.452611 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.452615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.453560 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.453574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.454659 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.454663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.454857 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.455709 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:18.455746 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:18.458498 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:18.462732 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.462737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.462740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.463364 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.463790 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.463794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.463796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.463935 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.464554 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:18.464557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:18.464700 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 02:27:18.464813 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:27:18.465065 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 02:27:18.465826 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.449644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.9.231.125:39197)
0620 02:27:18.449863 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.449869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.449893 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:27:18.450009 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.450014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.450114 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.450225 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.451294 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.452424 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.453751 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.454467 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.454626 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.458773 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.458781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.458785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.458829 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.462490 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.463498 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.463504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.463508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.463671 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.464051 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.464056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.464059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.464443 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.464619 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.464966 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.465244 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:27:18.466910 665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:27:18.467975 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.467595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56744)
0620 02:27:18.467764 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.467841 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:27:18.469022 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.468692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56758)
0620 02:27:18.468851 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.468929 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 02:27:18.469244 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.520529 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 1
I20260620 02:27:18.521826 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.521318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56772)
0620 02:27:18.521540 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.521646 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:27:18.523190 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.522727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56788)
0620 02:27:18.522898 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.522979 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 02:27:18.523458 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.551252 739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 3
I20260620 02:27:18.552505 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.551996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56802)
0620 02:27:18.552224 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.552329 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 02:27:18.553666 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.553201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56816)
0620 02:27:18.553450 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.553533 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260620 02:27:18.553872 739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.591741 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 2
I20260620 02:27:18.593031 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.592496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56822)
0620 02:27:18.592755 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.592853 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 02:27:18.594178 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.593781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56836)
0620 02:27:18.593953 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.594057 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:27:18.594408 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.697103 739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 4
I20260620 02:27:18.698621 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.698073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56838)
0620 02:27:18.698329 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.698419 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:27:18.699781 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.699372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56842)
0620 02:27:18.699547 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.699633 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 02:27:18.699978 739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
I20260620 02:27:18.700627 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 3
I20260620 02:27:18.701695 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.701192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56848)
0620 02:27:18.701398 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.701500 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 02:27:18.702852 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.702416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56854)
0620 02:27:18.702624 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.702714 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:27:18.703158 665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.231.124:38879: Network error: Client connection negotiation failed: client connection to 127.9.231.124:38879: connect: Connection refused (error 111)
W20260620 02:27:18.708612 747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:27:18.708997 747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:27:18.709100 747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:27:18.709270 747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:27:18.717147 747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:27:18.717259 747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:27:18.717346 747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:27:18.717430 747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:27:18.717545 747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:27:18.728585 747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.231.84:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.231.126:38395,127.9.231.125:39197,127.9.231.124:38879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/krb5kdc/kudu_127.9.231.124.keytab
--principal=kudu/127.9.231.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.231.124:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.231.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:27:18.730114 747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:27:18.731598 747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:27:18.746577 752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:18.746655 753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:27:18.747617 755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:27:18.749336 747 server_base.cc:1034] running on GCE node
I20260620 02:27:18.750355 747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:27:18.752115 747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:27:18.753367 747 hybrid_clock.cc:648] HybridClock initialized: now 1781922438753300 us; error 42 us; skew 500 ppm
I20260620 02:27:18.753768 747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:27:18.757654 747 init.cc:377] Logged in from keytab as kudu/127.9.231.124@KRBTEST.COM (short username kudu)
I20260620 02:27:18.759814 747 webserver.cc:460] Webserver started at http://127.9.231.124:39447/ using document root <none> and password file <none>
I20260620 02:27:18.760953 747 fs_manager.cc:359] Metadata directory not provided
I20260620 02:27:18.761107 747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:27:18.761458 747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:27:18.764766 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data/instance:
uuid: "a26112fa3020414fb0b548c6d26cff6e"
format_stamp: "Formatted at 2026-06-20 02:27:18 on dist-test-slave-9vg1"
server_key: "b574c36b426767c2f4fdc7738fdf4eaa"
server_key_iv: "6e23069429926a1303b601b900aefb8d"
server_key_version: "encryptionkey@0"
I20260620 02:27:18.765712 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/wal/instance:
uuid: "a26112fa3020414fb0b548c6d26cff6e"
format_stamp: "Formatted at 2026-06-20 02:27:18 on dist-test-slave-9vg1"
server_key: "b574c36b426767c2f4fdc7738fdf4eaa"
server_key_iv: "6e23069429926a1303b601b900aefb8d"
server_key_version: "encryptionkey@0"
I20260620 02:27:18.771454 747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:27:18.775329 762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:27:18.776808 747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:27:18.777012 747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data,/tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/wal
uuid: "a26112fa3020414fb0b548c6d26cff6e"
format_stamp: "Formatted at 2026-06-20 02:27:18 on dist-test-slave-9vg1"
server_key: "b574c36b426767c2f4fdc7738fdf4eaa"
server_key_iv: "6e23069429926a1303b601b900aefb8d"
server_key_version: "encryptionkey@0"
I20260620 02:27:18.777271 747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:27:18.797365 747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:27:18.802065 747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:27:18.802390 747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:27:18.822847 747 rpc_server.cc:307] RPC server started. Bound to: 127.9.231.124:38879
I20260620 02:27:18.822883 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.231.124:38879 every 8 connection(s)
I20260620 02:27:18.824352 747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/data/info.pb
I20260620 02:27:18.827237 10141 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4D_H_y/build/asan/bin/kudu as pid 747
I20260620 02:27:18.827440 10141 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4D_H_y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922055280103-10141-0/minicluster-data/master-2/wal/instance
I20260620 02:27:18.827991 10141 external_mini_cluster.cc:1428] Setting key 9f5ee941684d4de8ded7ed59a5f56480
I20260620 02:27:18.829025 815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:27:18.838274 815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
I20260620 02:27:18.851922 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.830051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.9.231.124:38879)
0620 02:27:18.830883 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.830897 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.830940 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:27:18.832653 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.832677 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.833118 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.833554 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.835146 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.836529 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.837863 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.838592 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.838810 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.843822 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.843830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.843841 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.843887 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.847112 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.847879 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.847889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.847905 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.848080 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.848503 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.848509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.848512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.849301 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.849747 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.850097 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.850591 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":366,"threads_started":1}
I20260620 02:27:18.858958 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.841243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.126:38395 (local address 127.0.0.1:34388)
0620 02:27:18.842305 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.842351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:18.842567 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:18.843090 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:18.843101 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:18.843786 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:18.844103 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.844114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.845282 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.845286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.846465 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.846478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.847313 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.847320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.847507 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.848161 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:18.848184 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:18.850754 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:18.854606 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.854619 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.854626 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.855339 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.856633 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.856639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.856643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.856859 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.857543 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:18.857552 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:18.857808 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 02:27:18.858031 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 02:27:18.858431 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":643,"threads_started":1}
I20260620 02:27:18.860155 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.841286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.9.231.126:38395)
0620 02:27:18.841947 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.841956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.842467 (+ 511us) server_negotiation.cc:407] Connection header received
0620 02:27:18.842754 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.842762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.842865 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.842981 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.844265 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.845096 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.846637 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.847112 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.847274 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.850997 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.851005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.851009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.851155 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.854374 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.855539 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.855577 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.855584 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.855950 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.857022 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.857028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.857047 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.857374 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.857503 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.858385 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.858889 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":372,"threads_started":1}
I20260620 02:27:18.862167 815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.126@KRBTEST.COM
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:27:18.876252 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.855293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.9.231.126:38395)
0620 02:27:18.856425 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.856435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.856472 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:27:18.856541 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.856546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.856633 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.856762 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.858426 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.859949 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.862121 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.862760 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.862914 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.869498 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.869504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.869507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.869543 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.872543 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.873949 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.873958 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.873962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.874166 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.874619 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.874625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.874633 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.874932 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.875267 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.875593 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.875827 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":733,"threads_started":1}
I20260620 02:27:18.878955 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.863359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.125:39197 (local address 127.0.0.1:44992)
0620 02:27:18.863675 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.863700 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:18.863924 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:18.864445 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:18.864451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:18.865004 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:18.865242 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.865257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.866990 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.866996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.868543 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.868560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.869323 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.869328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.869555 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.870257 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:18.870283 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:18.872560 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:18.876041 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.876047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.876051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.876563 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.876956 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.876961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.876965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.877141 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.877950 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:18.877958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:18.878213 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 02:27:18.878372 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 02:27:18.878704 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260620 02:27:18.879490 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.863225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.9.231.125:39197)
0620 02:27:18.863498 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.863506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.863801 (+ 295us) server_negotiation.cc:407] Connection header received
0620 02:27:18.864115 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.864121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.864232 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.864387 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.865437 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.866763 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.868707 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.869133 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.869241 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.872801 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.872807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.872811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.872849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.875889 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.876696 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.876701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.876703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.876839 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.877381 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.877389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.877394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.877754 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.877961 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.878525 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.878901 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:27:18.880419 665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 4
I20260620 02:27:18.880666 815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.126@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.124@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
Jun 20 02:27:18 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.231.125@KRBTEST.COM
I20260620 02:27:18.900748 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.881333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56870)
0620 02:27:18.882036 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.882061 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:18.882176 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:18.883033 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:18.883037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:18.883537 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:18.883879 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.883893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.885697 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.885702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.887349 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.887367 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.888663 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.888669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.888883 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.889823 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:18.889848 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:18.893358 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:18.897770 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.897775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.897780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.898307 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.898663 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.898668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.898672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.898806 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.899583 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:18.899587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:18.899770 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 02:27:18.899921 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 02:27:18.900337 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582}
I20260620 02:27:18.900859 823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.882143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.9.231.124:38879)
0620 02:27:18.882792 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.882795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.882812 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:27:18.882847 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.882850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.882898 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.883015 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.884086 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.885561 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.887557 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.888479 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.891554 (+ 3075us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.893611 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.893616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.893619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.893651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.897636 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.898416 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.898422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.898425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.898550 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.898916 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.898921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.898924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.899443 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.899605 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.900123 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.900468 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":233,"server-negotiator.queue_time_us":564,"spinlock_wait_cycles":31488,"thread_start_us":395,"threads_started":1}
I20260620 02:27:18.901363 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.881155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56868)
0620 02:27:18.881479 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:18.881499 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:18.881636 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:18.882590 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:18.882595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:18.883358 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:18.883658 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.883674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.885667 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.885675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.887349 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:18.887368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.888864 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.888871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:18.889127 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.889734 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:18.889767 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:18.893583 (+ 3816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:18.897306 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.897312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.897315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.897964 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.899372 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:18.899378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:18.899381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:18.899593 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:18.900320 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:18.900327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:18.900519 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 02:27:18.900672 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 02:27:18.901000 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260620 02:27:18.902391 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.881927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.9.231.124:38879)
0620 02:27:18.882089 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.882096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.882118 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:27:18.882168 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.882173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.882257 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.882368 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.883896 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.885377 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.887557 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.888479 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.888618 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.893876 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.893889 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.893894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.893946 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.897096 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.898965 (+ 1869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.898977 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.898982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.899213 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.899711 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.899715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.899720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.900044 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.900231 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.901675 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.901994 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":45}
I20260620 02:27:18.905516 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.878332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.9.231.125:39197)
0620 02:27:18.879826 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:18.879835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:18.879859 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:27:18.879929 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:18.879937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:18.880035 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:18.880163 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:18.882368 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.885679 (+ 3311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.888480 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:18.889496 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:18.889674 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:18.897109 (+ 7435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:18.897119 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:18.897123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:18.897175 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:18.900861 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.902144 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.902151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.902155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.902316 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:18.902903 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:18.902912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:18.902916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:18.903355 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:18.903657 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:18.904885 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0620 02:27:18.905143 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":12389,"threads_started":1}
I20260620 02:27:18.907567 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: Bootstrap starting.
I20260620 02:27:18.907680 665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642: Bootstrap starting.
I20260620 02:27:18.911558 815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:18.911583 665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:18.912878 10141 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:27:18.912969 665 log.cc:824] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:18.912971 815 log.cc:824] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:18.916086 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: No bootstrap required, opened a new log
I20260620 02:27:18.917145 639 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0a62608587c54c2bb244ee7ecd6e0642"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34392
I20260620 02:27:18.923026 665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642: No bootstrap required, opened a new log
I20260620 02:27:18.923880 815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
I20260620 02:27:18.924302 815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26112fa3020414fb0b548c6d26cff6e, State: Initialized, Role: FOLLOWER
I20260620 02:27:18.924889 815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
I20260620 02:27:18.925938 826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:18.926445 826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:18.927417 815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:18.929963 665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
I20260620 02:27:18.930390 665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a62608587c54c2bb244ee7ecd6e0642, State: Initialized, Role: FOLLOWER
I20260620 02:27:18.930874 665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
I20260620 02:27:18.931838 829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:18.932253 829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:18.933099 665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:18.933348 639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:27:18.933672 639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:18.938647 639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
I20260620 02:27:18.941795 639 leader_election.cc:288] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [CANDIDATE]: Term 1 election: Requested vote from peers 2739e08f46fc4b9bb4b7760392b0d1b1 (127.9.231.125:39197), a26112fa3020414fb0b548c6d26cff6e (127.9.231.124:38879)
I20260620 02:27:18.942628 790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a62608587c54c2bb244ee7ecd6e0642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a26112fa3020414fb0b548c6d26cff6e"
I20260620 02:27:18.943233 790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:27:18.946579 841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:27:18.946733 841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:27:18.946966 790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a62608587c54c2bb244ee7ecd6e0642 in term 1.
I20260620 02:27:18.948980 613 leader_election.cc:302] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a62608587c54c2bb244ee7ecd6e0642, a26112fa3020414fb0b548c6d26cff6e; no voters:
I20260620 02:27:18.949687 829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:27:18.966930 829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [term 1 LEADER]: Becoming Leader. State: Replica: 0a62608587c54c2bb244ee7ecd6e0642, State: Running, Role: LEADER
I20260620 02:27:18.967748 829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
W20260620 02:27:18.969163 618 tablet.cc:2369] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:27:18.951326 597 master_runner.cc:186] Error getting master registration for 127.9.231.125:39197: OK, instance_id { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" instance_seqno: 1781922438262834 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:27:18.975693 597 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:27:18.978430 844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a62608587c54c2bb244ee7ecd6e0642. Latest consensus state: current_term: 1 leader_uuid: "0a62608587c54c2bb244ee7ecd6e0642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:18.979569 844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:18.993184 850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:27:18.999185 739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } attempt: 5
I20260620 02:27:18.999660 850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 02:27:19 dist-test-slave-9vg1 krb5kdc[584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.231.125@KRBTEST.COM for kudu/127.9.231.124@KRBTEST.COM
I20260620 02:27:19.031014 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:19.000063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.9.231.124:38879)
0620 02:27:19.000283 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.000291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.000581 (+ 290us) server_negotiation.cc:407] Connection header received
0620 02:27:19.000657 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.000663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.000751 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.000851 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.002197 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.003415 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.005229 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.005961 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.006109 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:19.020220 (+ 14111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:19.020238 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:19.020245 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:19.020324 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:19.024909 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.026502 (+ 1593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.026508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.026512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.026709 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.028032 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.028038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.028041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.028470 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:19.028629 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:19.030343 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0620 02:27:19.030628 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:27:19.032941 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:18.999983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.231.124:38879 (local address 127.0.0.1:56878)
0620 02:27:19.000269 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 02:27:19.000292 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:27:19.000425 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:27:19.001058 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:27:19.001063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:27:19.001720 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:27:19.001982 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:19.001995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.003597 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.003602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:19.005024 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:27:19.005041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.006404 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.006411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:27:19.006665 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:19.007648 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:27:19.007708 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:27:19.019177 (+ 11469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:27:19.025311 (+ 6134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:19.025319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:19.025326 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:19.026290 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:19.027098 (+ 808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:27:19.027107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:27:19.027111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:27:19.027696 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:27:19.028921 (+ 1225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:27:19.028928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:27:19.029642 (+ 714us) client_negotiation.cc:769] Sending connection context
0620 02:27:19.031580 (+ 1938us) client_negotiation.cc:240] Negotiation successful
0620 02:27:19.032177 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260620 02:27:19.033680 860 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:27:19.041692 739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: Bootstrap starting.
W20260620 02:27:19.054804 768 tablet.cc:2369] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:27:19.060003 739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: Neither blocks nor log segments found. Creating new log.
I20260620 02:27:19.061393 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:19.011947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.9.231.126:38395)
0620 02:27:19.012192 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.012201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.023032 (+ 10831us) server_negotiation.cc:407] Connection header received
0620 02:27:19.023416 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.023423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.023538 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.023617 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.026188 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.027401 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.029448 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.030193 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.030315 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:19.050404 (+ 20089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:19.050415 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:19.050423 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:19.050490 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:19.055480 (+ 4990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.056666 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.056675 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.056681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.056888 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.057548 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.057582 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.057588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.058094 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:19.058427 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:19.060458 (+ 2031us) server_negotiation.cc:299] Negotiation successful
0620 02:27:19.060821 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:27:19.045851 747 master_runner.cc:186] Error getting master registration for 127.9.231.125:39197: OK, instance_id { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" instance_seqno: 1781922438262834 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:27:19.061753 747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:27:19.061086 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:19.011278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.9.231.125:39197)
0620 02:27:19.011536 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.011546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.019938 (+ 8392us) server_negotiation.cc:407] Connection header received
0620 02:27:19.020723 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.020732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.020847 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.020937 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.031868 (+ 10931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.033345 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.048263 (+ 14918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.049040 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.049178 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:19.052792 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:19.052802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:19.052807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:19.052857 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:19.056709 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.058109 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.058115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.058119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.058287 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.058748 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.058753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.058756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.059176 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:19.059430 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:19.060412 (+ 982us) server_negotiation.cc:299] Negotiation successful
0620 02:27:19.060686 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:27:19.070787 739 log.cc:824] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:27:19.074296 790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [term 1 FOLLOWER]: Refusing update from remote peer 0a62608587c54c2bb244ee7ecd6e0642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:27:19.075870 829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:27:19.080901 823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:19.022198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.9.231.124:38879)
0620 02:27:19.022489 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.022498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.061821 (+ 39323us) server_negotiation.cc:407] Connection header received
0620 02:27:19.062102 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.062109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.062201 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.062296 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.065770 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.066924 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.068856 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:19.069507 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:19.069616 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:19.073294 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:27:19.073302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:27:19.073307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:27:19.073357 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:27:19.077174 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.078228 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.078234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.078238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.078393 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:27:19.078901 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:27:19.078906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:27:19.078910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:27:19.079378 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:27:19.079647 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:19.080252 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 02:27:19.080510 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:27:19.090083 739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: No bootstrap required, opened a new log
I20260620 02:27:19.092657 826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a62608587c54c2bb244ee7ecd6e0642. Latest consensus state: current_term: 1 leader_uuid: "0a62608587c54c2bb244ee7ecd6e0642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:19.093016 826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:19.097584 844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a62608587c54c2bb244ee7ecd6e0642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:19.097935 844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:19.099308 850 catalog_manager.cc:1347] Generated new cluster ID: 5ccce05cb8b948fba40cebee229fb033
I20260620 02:27:19.099427 850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:27:19.106129 739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
I20260620 02:27:19.106581 739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2739e08f46fc4b9bb4b7760392b0d1b1, State: Initialized, Role: FOLLOWER
I20260620 02:27:19.107110 739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } }
I20260620 02:27:19.114225 867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:19.114708 867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:19.116700 739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:27:19.117182 713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:27:19.121287 713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [term 1 FOLLOWER]: Refusing update from remote peer 0a62608587c54c2bb244ee7ecd6e0642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:27:19.121941 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a62608587c54c2bb244ee7ecd6e0642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1"
I20260620 02:27:19.123085 829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:27:19.125994 714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a62608587c54c2bb244ee7ecd6e0642 in term 1.
I20260620 02:27:19.130506 826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a62608587c54c2bb244ee7ecd6e0642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:19.130991 826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:19.188687 867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a62608587c54c2bb244ee7ecd6e0642. Latest consensus state: current_term: 1 leader_uuid: "0a62608587c54c2bb244ee7ecd6e0642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:19.188678 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a62608587c54c2bb244ee7ecd6e0642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:19.189222 867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:19.189224 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:27:19.200507 829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a62608587c54c2bb244ee7ecd6e0642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a62608587c54c2bb244ee7ecd6e0642" member_type: VOTER last_known_addr { host: "127.9.231.126" port: 38395 } } peers { permanent_uuid: "2739e08f46fc4b9bb4b7760392b0d1b1" member_type: VOTER last_known_addr { host: "127.9.231.125" port: 39197 } } peers { permanent_uuid: "a26112fa3020414fb0b548c6d26cff6e" member_type: VOTER last_known_addr { host: "127.9.231.124" port: 38879 } } }
I20260620 02:27:19.200943 829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642 [sys.catalog]: This master's current role is: LEADER
I20260620 02:27:19.210144 882 catalog_manager.cc:1259] Loaded cluster ID: 5ccce05cb8b948fba40cebee229fb033
I20260620 02:27:19.210259 882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: loading cluster ID for follower catalog manager: success
W20260620 02:27:19.211596 882 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 02:27:19.211740 882 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:27:19.213245 850 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:27:19.216032 850 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:27:19.250564 850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a62608587c54c2bb244ee7ecd6e0642: Generated new TSK 0
I20260620 02:27:19.252235 850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:27:19.343102 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.336439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.9.231.125:39197)
0620 02:27:19.336692 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.336707 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.339839 (+ 3132us) server_negotiation.cc:407] Connection header received
0620 02:27:19.340214 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.340226 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.340331 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.340437 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.342689 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45022: BlockingRecv error: recv got EOF from 127.0.0.1:45022 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 02:27:19.345172 821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.335802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.9.231.126:38395)
0620 02:27:19.336079 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.336087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.339766 (+ 3679us) server_negotiation.cc:407] Connection header received
0620 02:27:19.343437 (+ 3671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.343444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.343532 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.343608 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.344899 (+ 1291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34420: BlockingRecv error: recv got EOF from 127.0.0.1:34420 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 02:27:19.346607 823 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.340819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.9.231.124:38879)
0620 02:27:19.341075 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.341085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.344139 (+ 3054us) server_negotiation.cc:407] Connection header received
0620 02:27:19.344585 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.344595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.344717 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.344821 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.346212 (+ 1391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56892: BlockingRecv error: recv got EOF from 127.0.0.1:56892 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 02:27:19.361770 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.352431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.9.231.125:39197)
0620 02:27:19.352688 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.352698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.359768 (+ 7070us) server_negotiation.cc:407] Connection header received
0620 02:27:19.360060 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.360065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.360174 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.360310 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.361415 (+ 1105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45024: BlockingRecv error: recv got EOF from 127.0.0.1:45024 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 02:27:19.374109 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.371728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.9.231.125:39197)
0620 02:27:19.371974 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.371984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.372568 (+ 584us) server_negotiation.cc:407] Connection header received
0620 02:27:19.372822 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.372827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.372927 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.373049 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.373841 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45036: BlockingRecv error: recv got EOF from 127.0.0.1:45036 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 02:27:19.382055 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.379552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.9.231.125:39197)
0620 02:27:19.379825 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.379834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.380367 (+ 533us) server_negotiation.cc:407] Connection header received
0620 02:27:19.380646 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.380652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.380766 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.380879 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.381774 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45038: BlockingRecv error: recv got EOF from 127.0.0.1:45038 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260620 02:27:19.392230 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.389662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.9.231.125:39197)
0620 02:27:19.389992 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.390002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.390603 (+ 601us) server_negotiation.cc:407] Connection header received
0620 02:27:19.390828 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.390840 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.390920 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.391040 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.391983 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45052: BlockingRecv error: recv got EOF from 127.0.0.1:45052 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
W20260620 02:27:19.407500 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.404694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.9.231.125:39197)
0620 02:27:19.404956 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.404966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.405714 (+ 748us) server_negotiation.cc:407] Connection header received
0620 02:27:19.405969 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.405974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.406058 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.406186 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.407224 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45068: BlockingRecv error: recv got EOF from 127.0.0.1:45068 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 02:27:19.431953 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.429356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.9.231.125:39197)
0620 02:27:19.429652 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.429662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.430332 (+ 670us) server_negotiation.cc:407] Connection header received
0620 02:27:19.430577 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.430583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.430697 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.430815 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.431667 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45080: BlockingRecv error: recv got EOF from 127.0.0.1:45080 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 02:27:19.470949 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.468347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.9.231.125:39197)
0620 02:27:19.468632 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.468648 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.469356 (+ 708us) server_negotiation.cc:407] Connection header received
0620 02:27:19.469633 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.469638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.469716 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.469834 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.470665 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45090: BlockingRecv error: recv got EOF from 127.0.0.1:45090 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 02:27:19.543381 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.541109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.9.231.125:39197)
0620 02:27:19.541318 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.541324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.541513 (+ 189us) server_negotiation.cc:407] Connection header received
0620 02:27:19.541862 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.541868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.541964 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.542101 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.543108 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45094: BlockingRecv error: recv got EOF from 127.0.0.1:45094 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 02:27:19.683526 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.681109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.9.231.125:39197)
0620 02:27:19.681368 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.681373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.681775 (+ 402us) server_negotiation.cc:407] Connection header received
0620 02:27:19.682047 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.682051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.682137 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.682228 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.683257 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45106: BlockingRecv error: recv got EOF from 127.0.0.1:45106 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 02:27:19.946640 744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:27:19.944694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.9.231.125:39197)
0620 02:27:19.944923 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:19.944931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:19.945159 (+ 228us) server_negotiation.cc:407] Connection header received
0620 02:27:19.945281 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:19.945285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:19.945393 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:19.945499 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:27:19.946422 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45116: BlockingRecv error: recv got EOF from 127.0.0.1:45116 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:27:20.038581 860 catalog_manager.cc:1259] Loaded cluster ID: 5ccce05cb8b948fba40cebee229fb033
I20260620 02:27:20.038694 860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: loading cluster ID for follower catalog manager: success
I20260620 02:27:20.041483 860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: acquiring CA information for follower catalog manager: success
I20260620 02:27:20.043637 860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a26112fa3020414fb0b548c6d26cff6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:27:20.215413 882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: acquiring CA information for follower catalog manager: success
I20260620 02:27:20.217730 882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2739e08f46fc4b9bb4b7760392b0d1b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:27:20.468953 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:20.462518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.9.231.125:39197)
0620 02:27:20.463038 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:20.463044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:20.463437 (+ 393us) server_negotiation.cc:407] Connection header received
0620 02:27:20.463699 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:20.463704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:20.463871 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:20.464016 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:20.464461 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.465189 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.466558 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.466987 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.467186 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:20.467719 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:20.468293 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:20.468462 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:20.468527 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 02:27:20.468635 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":275,"threads_started":1}
I20260620 02:27:20.483641 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:20.477107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.9.231.124:38879)
0620 02:27:20.477684 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:20.477690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:20.477813 (+ 123us) server_negotiation.cc:407] Connection header received
0620 02:27:20.477998 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:20.478002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:20.478185 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:20.478343 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:20.478804 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.479755 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.481185 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.481695 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.481792 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:20.482289 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:20.482843 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:20.482953 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:20.483193 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 02:27:20.483270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":297,"threads_started":1}
I20260620 02:27:20.498178 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:20.490230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.9.231.126:38395)
0620 02:27:20.491022 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:20.491029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:20.491053 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:27:20.491234 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:20.491242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:20.491484 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:20.491673 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:20.492499 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.493620 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.495683 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.496096 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.496250 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:20.496865 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:20.497437 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:20.497547 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:20.497698 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 02:27:20.497801 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":556,"threads_started":1}
I20260620 02:27:20.499264 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:20.491676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.9.231.124:38879)
0620 02:27:20.491841 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:20.491847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:20.492759 (+ 912us) server_negotiation.cc:407] Connection header received
0620 02:27:20.492934 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:20.492940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:20.493028 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:20.493121 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:20.493725 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.494785 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.496857 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.497517 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.497645 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:20.498187 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:20.498318 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:20.498406 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:20.498856 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 02:27:20.498935 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:27:20.499369 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:27:20.490309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.9.231.125:39197)
0620 02:27:20.490544 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:27:20.490550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:27:20.491527 (+ 977us) server_negotiation.cc:407] Connection header received
0620 02:27:20.491849 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:27:20.491855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:27:20.491932 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:27:20.492020 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:27:20.493012 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.494153 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.496457 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:27:20.497110 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:27:20.497241 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:27:20.498492 (+ 1251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:27:20.498619 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:27:20.498722 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:27:20.498910 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 02:27:20.499007 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 02:27:20.503952 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 597
I20260620 02:27:20.505234 660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:20.621861 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 597
I20260620 02:27:20.644291 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 672
I20260620 02:27:20.645485 734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:20.755919 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 672
I20260620 02:27:20.777954 10141 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task4D_H_y/build/asan/bin/kudu pid 747
I20260620 02:27:20.779247 810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:27:20.892688 10141 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4D_H_y/build/asan/bin/kudu with pid 747
2026-06-20T02:27:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5062 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10250 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (385567 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST